Natural gas load prediction method based on improved Transform network

By improving the Transformer network, combined with the LSTM network and residual connection mechanism, the problems of insufficient accuracy and data complexity in natural gas load prediction are solved, which significantly improves the prediction accuracy and robustness, and meets the real-time prediction needs.

CN119026727APending Publication Date: 2024-11-26BEIFANG UNIV OF NATITIES
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411009964.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-26

Smart Images

  • Figure CN119026727A_ABST
    Figure CN119026727A_ABST
Patent Text Reader

Abstract

The invention discloses a natural gas load prediction method based on an improved Transform network, and the method comprises the steps: carrying out the preprocessing of collected natural gas load data and meteorological data, decomposing the preprocessed natural gas load data into a plurality of intrinsic mode functions IMF through employing a variational mode decomposition method VMD, and carrying out the prediction of the natural gas load data through employing the IMF; the method comprises the following steps of: preprocessing meteorological data, selecting a proper K value by using a minimum envelope entropy as a judgment basis, then independently splicing each IMF and the preprocessed meteorological data to form a plurality of pieces of data to be measured, normalizing the data to be measured, and matching each piece of data to be measured with a trained improved Transformer network to obtain a plurality of pieces of data to be measured; and inputting the plurality of normalized data to be measured into the respective corresponding improved Transform networks for prediction to obtain corresponding prediction results, and finally performing superposition reconstruction on all the prediction results to serve as a final prediction result to be output. According to the method, the problems of data non-stationarity and complex time sequence dependency relationship in natural gas load prediction are solved, and the prediction accuracy and reliability are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of natural gas load forecasting, and in particular to a natural gas load forecasting method based on an improved Transformer network. Background Art

[0002] In the field of natural gas load forecasting, traditional methods usually rely on statistical models and time series analysis methods. However, these methods have certain limitations when dealing with the non-stationarity and complex time series dependencies of natural gas load data, resulting in prediction accuracy that is difficult to meet actual needs. On the other hand, prediction methods based on machine learning and deep learning are also widely used in natural gas load forecasting, but due to problems such as complex model design and difficult data preprocessing, these methods still face many challenges in practical applications.

[0003] In recent years, algorithms represented by Transformer and LSTM networks have been applied to the field of natural gas load forecasting. Compared with traditional statistical models and time series analysis methods, deep learning methods have stronger robustness and better generalization ability, and can more effectively handle nonlinearity and complex time series dependencies in natural gas load data. In summary, designing a method with high-precision prediction capabilities is the focus of research in the field of natural gas load forecasting. Summary of the invention

[0004] The present invention aims to solve the problems of insufficient accuracy, large data fluctuations and complex time series dependencies faced in natural gas load forecasting. A natural gas load forecasting method based on an improved Transformer network is proposed, which can effectively deal with the prediction errors caused by data non-stationarity and temperature changes in natural gas load forecasting, and significantly improve the accuracy of the prediction.

[0005] To achieve the above purpose, the technical solution provided by the present invention is: a natural gas load forecasting method based on an improved Transformer network, wherein the improved Transformer network is an improvement on the decoder of the traditional Transformer network, the original decoder structure is abandoned, and an LSTM network and a fully connected layer are used to replace it, so as to construct a new decoder structure, and at the same time, a residual connection mechanism is introduced and integrated into the encoder and the new decoder structure of the Transformer network;

[0006] The specific implementation of the natural gas load forecasting method includes the following steps:

[0007] 1) Collect data, including meteorological data and natural gas load data, where meteorological data includes temperature, humidity, air pressure and rainfall;

[0008] 2) Preprocess the collected data to ensure the continuity of the data;

[0009] 3) The preprocessed natural gas load data is decomposed using the variational mode decomposition method VMD to obtain several intrinsic mode functions IMF. At the same time, the minimum envelope entropy is used as the judgment basis to select the appropriate K value, where the envelope entropy is used to measure the complexity and uncertainty of the signal. The smaller the entropy value, the lower the complexity and uncertainty of the signal.

[0010] 4) Each IMF is separately spliced ​​with the preprocessed meteorological data to form multiple test data and normalized. Each test data is paired with a trained improved Transformer network. The normalized multiple test data are respectively input into their respective improved Transformer networks for prediction to obtain the corresponding prediction results. Finally, all the prediction results are superimposed and reconstructed as the final prediction results for output.

[0011] Further, in step 2), the collected data is preprocessed, including:

[0012] Identify and process missing values ​​in the data and use the same-period filling method to supplement them;

[0013] Detect and delete outliers, and visually observe and discover data points that deviate from the overall trend by drawing scatter plots and line graphs.

[0014] Further, in step 3), the preprocessed natural gas load data is decomposed using the variational mode decomposition method VMD, and the specific process is as follows:

[0015] First, initialize the VMD parameters, including the number of modes K, the center frequency of each mode ω k , modal function u k , penalty parameter α, tolerance ε and Lagrange multiplier λ;

[0016] Secondly, iteratively update the modal function u k , center frequency ω k and the Lagrange multiplier λ:

[0017]

[0018] In the formula, represents the modal function u at the n+1th iteration k , represents the sum of all modes except the kth mode at the n+1th iteration, where i is the index and ω represents the frequency, represents the center frequency of the kth mode function at the nth iteration, represents the center frequency of the kth mode function at the n+1th iteration, represents the time-weighted energy integral of the kth mode function, represents the total energy integral of the kth mode function, λ n represents the Lagrange multiplier at the nth iteration, λ n+1 represents the Lagrange multiplier at the n+1th iteration, τ represents the step size parameter used to adjust the update speed, f is the natural gas load sequence, represents the sum of all modal functions;

[0019] Next, determine whether the convergence condition is met: if the update amount of the modal function is less than the set tolerance ε, stop the iteration; otherwise, return to continue the iteration, as shown below:

[0020]

[0021] In the formula, represents the modal function u at the nth iteration k , represents the bi-norm, ε is the preset tolerance used to judge convergence;

[0022] Finally, the envelope entropy of each modal function is calculated, and all envelope entropies are added to obtain the total envelope entropy. The total envelope entropy under different K values ​​is compared, and the K value corresponding to the minimum total envelope entropy is selected as the final decomposition mode number, which is expressed as follows:

[0023]

[0024] In the formula, E k represents the envelope entropy of the kth IMF, A k (m) represents the instantaneous envelope of the kth IMF at the mth moment, M represents the length of the natural gas load sequence, that is, the number of data points contained in the IMF, and E total represents the total envelope entropy of all IMFs.

[0025] Furthermore, in step 4), the IMF and the preprocessed meteorological data are combined in the order of the time series to form a test data, and the test data is normalized using MinMaxScaler to scale the data to the interval [0,1] to eliminate the dimensional difference between the data, where the normalization formula is as follows:

[0026]

[0027] In the formula, x is the original data, x' is the normalized data, and x min is the minimum value in the data to be tested, x max is the maximum value in the data to be tested.

[0028] Furthermore, in step 4), the normalized test data is input into the improved Transformer network, and the position information is first embedded through the position encoding formula. The calculation formula is as follows:

[0029]

[0030] In the formula, pos represents the position, index represents the dimension index, and d model Indicates dimension, PE (pos,2index) Represents the sine value of the 2nd index dimension at position pos, PE (pos,2index+1) Represents the cosine value of the 2ndindex+1th dimension at position pos;

[0031] The time series with embedded position information is input into the encoder, which contains a multi-head attention mechanism Multi-Head Attention and a feed-forward neural network layer to capture global dependencies and local patterns in the sequence. Multi-Head Attention plays a key role in achieving accurate prediction. Each attention head can capture information in different representation subspaces by learning different weights. The outputs of multiple attention heads are then linearly transformed and connected together. The calculation formula is as follows:

[0032] MultiHead=Concat(head1,head2,…,head j ,…,head n′ )*W o

[0033] In the formula, MultiHead represents multi-head attention, Concat(·) represents connecting the outputs of each head, and head n′ represents the n′th attention head, W o represents the output weight matrix;

[0034] In Multi-Head Attention, each attention head performs the calculation process of the self-attention mechanism, thereby focusing on the internal information of the sequence. The calculation formula is as follows:

[0035]

[0036] In the formula, Query represents the input of the current position, which is used to calculate the attention weight of this position and other positions; Key represents the input of other positions, which is used to match the query and calculate the attention weight; Value represents the input of other positions, which is used to generate the output of the current position according to the weighted result of the attention weight; denote the linear transformation matrices for query, key, and value, respectively; Attention(·) denotes the computation of the self-attention mechanism;

[0037] Next, the extracted feature information is fed into the decoder to generate a preliminary prediction result; wherein the decoder is composed of an LSTM network and a fully connected layer, as follows:

[0038] The LSTM network uses three key mechanisms, namely the forget gate, the input gate and the output gate, to regulate the flow of information, thereby selectively retaining, discarding and updating information;

[0039] The forget gate will be based on the current input x t and the hidden state h at the previous moment t-1 And the corresponding weight matrix W f and the bias term b f To calculate a value f between 0 and 1 t , represents the amount of information that needs to be retained, and is converted into the following formula:

[0040] f t =sigmoid(W f .[h t-1 ,x t ]+b f )

[0041] The input gate first determines which new information should be updated or written through the sigmoid activation function, and then creates a candidate vector through the tanh activation function To update the cell state; the input gate will be based on the current input x t and the hidden state h at the previous moment t-1 And the corresponding weight matrix W i , W c and the bias term b i 、b c Calculate a value i between 0 and 1 t , represents the degree to which the input gate is to be updated, and a candidate vector The converted formula is as follows:

[0042]

[0043] In the formula, C t-1 Indicates the cell state at the last moment, C t Indicates the cell state at the current moment;

[0044] The output gate first determines the cell state C through the sigmoid activation function tWhich parts of the cell state will be passed out, and then the value of the cell state is converted to between -1 and 1 through the tanh activation function, and finally the output of the tanh activation is multiplied element by element with the output of the output gate to generate the final output value; the output gate will be based on the current input x t and the hidden state h at the previous moment t-1 And the corresponding weight matrix W o and the bias term b o Calculate a value O between 0 and 1 t , represents the amount of information to be transmitted by the output gate, which is converted into the following formula:

[0045]

[0046] In the formula, h t Indicates the hidden state at the current moment;

[0047] And a residual connection mechanism is introduced between the encoder and the LSTM network. The input is directly added to the output through the residual connection mechanism. The calculation formula is as follows:

[0048] output = dropout(LSTM(x input ))+x input

[0049] In the formula, output represents the preliminary prediction result, and dropout(·) randomly sets the output of some neurons to zero, thereby reducing the network's dependence on specific neurons and preventing the network from overfitting. input Represents the features extracted by the encoder;

[0050] The output is input into the fully connected layer for re-prediction, and the result is used as the prediction result of the improved Transformer network; in the fully connected layer, each neuron is connected to all neurons in the previous layer, and an activation function ReLU is connected after each fully connected layer;

[0051] Finally, the prediction results of each improved Transformer network are superimposed to form the final prediction result.

[0052] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0053] 1. Use the variational mode decomposition method VMD to decompose the preprocessed natural gas load data to obtain several intrinsic mode functions IMF. At the same time, the minimum envelope entropy is used as the judgment basis to select the appropriate K value, where the envelope entropy is used to measure the complexity and uncertainty of the signal. The smaller the entropy value, the lower the complexity and uncertainty of the signal. This method can effectively reduce the non-stationarity of the data and reduce noise interference, making the data more stable and pure. This provides a more reliable basis for subsequent feature extraction and prediction.

[0054] 2. Improve the decoder structure of the traditional Transformer network. Abandon the original decoder design, use LSTM network and fully connected layer to replace it, build a new decoder structure, and introduce the residual connection mechanism and integrate it into the encoder and new decoder structure of the Transformer network. This improvement has the following advantages:

[0055] 2.1. Improve the network's time series processing capabilities. LSTM networks are good at processing time series data and can better capture the time series information in the data. Compared with traditional Transformer decoders, LSTM networks can more effectively remember and utilize historical information, thereby improving the network's performance when processing long-span data.

[0056] 2.2. Enhance the generalization ability of the network. The fully connected layer can adjust the network parameters more flexibly to adapt to different data distributions and feature structures. With the introduction of the fully connected layer, the network can better adapt to different application scenarios and improve its generalization ability.

[0057] 2.3. Introducing the residual connection mechanism, which can alleviate the gradient vanishing and gradient exploding problems in deep networks, and ensure smoother information transmission in deep structures. This not only improves the training efficiency of the network, but also improves the stability and prediction accuracy of the network.

[0058] 2.4. Improve prediction accuracy. The improved decoder structure combined with the LSTM network and the fully connected layer can more accurately capture the complex relationships and nonlinear features in the data, thereby improving prediction accuracy.

[0059] 2.5. To meet the needs of real-time prediction, the improved Transformer network not only improves accuracy, but also its structural optimization enables the model to meet the needs of real-time prediction in practical applications. By introducing the residual connection mechanism and optimizing the decoder structure, the model has higher efficiency and robustness when processing large-scale data. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1To improve the structure diagram of the Transformer network; the feature blocks in the figure include temperature, humidity, rainfall and air pressure data, the IMF block is the intrinsic mode function after variational mode decomposition, Concat represents the splicing operation, the normalization block represents normalizing the data to the range of [0,1], and the reconstruction superposition block represents adding all the prediction results to obtain the final prediction value.

[0061] Figure 2 This is the flow chart of minimum envelope entropy and variational mode decomposition.

[0062] Figure 3 This is the structural diagram of the Multi-Head Attention mechanism; in the figure, Query, Key, and Value are the query, key value, and weight of the feature graph vector, respectively. DETAILED DESCRIPTION

[0063] The present invention will be further described below in conjunction with specific examples, but the embodiments of the present invention are not limited thereto.

[0064] This embodiment discloses a natural gas load forecasting method based on an improved Transformer network. The improved Transformer network is an improvement on the decoder of a traditional Transformer network. The improved Transformer network architecture is as follows: Figure 1 As shown in the figure, the original decoder structure is abandoned and replaced by LSTM network and fully connected layer to construct a new decoder structure. At the same time, the residual connection mechanism is introduced and integrated into the encoder and new decoder structure of the Transformer network.

[0065] The specific implementation of the natural gas load forecasting method comprises the following steps:

[0066] 1) Collect data, including meteorological data and natural gas load data, where meteorological data includes temperature, humidity, air pressure and rainfall.

[0067] 2) Preprocess the collected data to ensure the continuity of the data, including:

[0068] Identify and process missing values ​​in the data and use the same-period filling method to supplement them;

[0069] Detect and delete outliers, and visually observe and discover data points that deviate from the overall trend by drawing scatter plots and line graphs.

[0070] 3) The preprocessed natural gas load data is decomposed using the variational mode decomposition method VMD to obtain several intrinsic mode functions IMF. At the same time, the minimum envelope entropy is used as the judgment basis to select the appropriate K value, where the envelope entropy is used to measure the complexity and uncertainty of the signal. The smaller the entropy value, the lower the complexity and uncertainty of the signal. The decomposition process is as follows: Figure 2 As shown;

[0071] The variational mode decomposition method VMD is used to decompose the preprocessed natural gas load data. The specific process is as follows:

[0072] First, initialize the VMD parameters, including the number of modes K, the center frequency of each mode ω k , modal function u k , penalty parameter α, tolerance ε and Lagrange multiplier λ;

[0073] Secondly, iteratively update the modal function u k , center frequency ω k and the Lagrange multiplier λ:

[0074]

[0075] In the formula, represents the modal function u at the n+1th iteration k , represents the sum of all modes except the kth mode at the n+1th iteration, where i is the index and ω represents the frequency, represents the center frequency of the kth mode function at the nth iteration, represents the center frequency of the kth mode function at the n+1th iteration, represents the time-weighted energy integral of the kth mode function, represents the total energy integral of the kth mode function, λ n represents the Lagrange multiplier at the nth iteration, λ n+1 represents the Lagrange multiplier at the n+1th iteration, τ represents the step size parameter used to adjust the update speed, f is the natural gas load sequence, represents the sum of all modal functions;

[0076] Next, determine whether the convergence condition is met: if the update amount of the modal function is less than the set tolerance ε, stop the iteration; otherwise, return to continue the iteration, as shown below:

[0077]

[0078] In the formula, represents the modal function u at the nth iteration k , represents the bi-norm, ε is the preset tolerance used to judge convergence;

[0079] Finally, the envelope entropy of each modal function is calculated, and all envelope entropies are added to obtain the total envelope entropy. The total envelope entropy under different K values ​​is compared, and the K value corresponding to the minimum total envelope entropy is selected as the final decomposition mode number, which is expressed as follows:

[0080]

[0081] In the formula, E k represents the envelope entropy of the kth IMF, A k (m) represents the instantaneous envelope of the kth IMF at the mth moment, M represents the length of the natural gas load sequence, that is, the number of data points contained in the IMF, and E total represents the total envelope entropy of all IMFs.

[0082] 4) Each IMF is separately spliced ​​with the preprocessed meteorological data to form multiple test data, and each test data is paired with a trained improved Transformer network. The multiple test data are respectively input into their respective improved Transformer networks for prediction to obtain the corresponding prediction results. Finally, all the prediction results are superimposed and reconstructed as the final prediction results for output.

[0083] Specifically, the IMF and the preprocessed meteorological data are combined in the order of time series to form a test data. The test data is normalized using MinMaxScaler to scale the data to the interval [0,1] to eliminate the dimensional differences between the data. The normalization formula is as follows:

[0084]

[0085] In the formula, x is the original data, x' is the normalized data, and x min is the minimum value in the data to be tested, x max is the maximum value in the data to be tested.

[0086] The normalized test data is input into the improved Transformer network. First, the position information is embedded through the position encoding formula. The calculation formula is as follows:

[0087]

[0088] In the formula, pos represents the position, index represents the dimension index, and d model Indicates dimension, PE (pos,2index) Represents the sine value of the 2nd index dimension at position pos, PE (pos,2index+1)Represents the cosine value of the 2ndindex+1th dimension at position pos;

[0089] The time series with embedded position information is input into the encoder, which contains a multi-head attention mechanism and a feed forward neural network layer to capture the global dependencies and local patterns in the sequence. The structure of the multi-head attention mechanism is as follows: Figure 3 As shown in the figure, Multi-Head Attention plays a key role in achieving accurate prediction. Each attention head can capture information in different representation subspaces by learning different weights. Then the outputs of multiple attention heads are linearly transformed and connected together. The calculation formula is as follows:

[0090] MultiHead=Concat(head1,head2,…,head j ,…,head n )*W o

[0091] In the formula, MultiHead represents multi-head attention, Concat(·) represents connecting the outputs of each head, and head n′ represents the n′th attention head, W o represents the output weight matrix;

[0092] In Multi-Head Attention, each attention head performs the calculation process of the self-attention mechanism, thereby focusing on the internal information of the sequence. The calculation formula is as follows:

[0093]

[0094] In the formula, Query represents the input of the current position, which is used to calculate the attention weight of this position and other positions; Key represents the input of other positions, which is used to match the query and calculate the attention weight; Value represents the input of other positions, which is used to generate the output of the current position according to the weighted result of the attention weight; denote the linear transformation matrices for query, key, and value, respectively; Attention(·) denotes the computation of the self-attention mechanism;

[0095] Next, the extracted feature information is fed into the decoder to generate a preliminary prediction result; wherein the decoder is composed of an LSTM network and a fully connected layer, as follows:

[0096] The LSTM network uses three key mechanisms, namely the forget gate, the input gate and the output gate, to regulate the flow of information, thereby selectively retaining, discarding and updating information;

[0097] The forget gate will be based on the current input x t and the hidden state h at the previous moment t-1 And the corresponding weight matrix W f and the bias term b f To calculate a value f between 0 and 1 t , represents the amount of information that needs to be retained, and is converted into the following formula:

[0098] f t =sigmoid(W f .[h t-1 ,x t ]+b f )

[0099] The input gate first determines which new information should be updated or written through the sigmoid activation function, and then creates a candidate vector through the tanh activation function To update the cell state; the input gate will be based on the current input x t and the hidden state h at the previous moment t-1 And the corresponding weight matrix W i , W c and the bias term b i 、b c Calculate a value i between 0 and 1 t , represents the degree to which the input gate is to be updated, and a candidate vector The converted formula is as follows:

[0100]

[0101] In the formula, C t-1 Indicates the cell state at the last moment, C t Indicates the cell state at the current moment;

[0102] The output gate first determines the cell state C through the sigmoid activation function t Which parts of the cell state will be passed out, and then the value of the cell state is converted to between -1 and 1 through the tanh activation function, and finally the output of the tanh activation is multiplied element by element with the output of the output gate to generate the final output value; the output gate will be based on the current input x t and the hidden state h at the previous moment t-1 And the corresponding weight matrix W o and the bias term b oCalculate a value O between 0 and 1 t , represents the amount of information to be transmitted by the output gate, which is converted into the following formula:

[0103]

[0104] In the formula, h t Indicates the hidden state at the current moment;

[0105] And a residual connection mechanism is introduced between the encoder and the LSTM network. The input is directly added to the output through the residual connection mechanism. The calculation formula is as follows:

[0106] output = dropout(LSTM(x input ))+x input

[0107] In the formula, output represents the preliminary prediction result, and dropout(·) randomly sets the output of some neurons to zero, thereby reducing the network's dependence on specific neurons and preventing the network from overfitting. input Represents the features extracted by the encoder;

[0108] The output is input into the fully connected layer for re-prediction, and the result is used as the prediction result of the improved Transformer network; in the fully connected layer, each neuron is connected to all neurons in the previous layer, and an activation function ReLU is connected after each fully connected layer;

[0109] Finally, the prediction results of each improved Transformer network are superimposed to form the final prediction result.

[0110] The above-described embodiments are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, all changes made according to the shape and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A natural gas load forecasting method based on an improved Transformer network, characterized in that: The improved Transformer network is an improvement on the decoder of the traditional Transformer network. The original decoder structure is abandoned and replaced by an LSTM network and a fully connected layer to construct a new decoder structure. At the same time, the residual connection mechanism is introduced and integrated into the encoder and new decoder structure of the Transformer network. The specific implementation of the natural gas load forecasting method includes the following steps: 1) Collect data, including meteorological data and natural gas load data, where meteorological data includes temperature, humidity, air pressure and rainfall; 2) Preprocess the collected data to ensure the continuity of the data; 3) The preprocessed natural gas load data is decomposed using the variational mode decomposition method VMD to obtain several intrinsic mode functions IMF. At the same time, the minimum envelope entropy is used as the judgment basis to select the appropriate K value, where the envelope entropy is used to measure the complexity and uncertainty of the signal. The smaller the entropy value, the lower the complexity and uncertainty of the signal. 4) Each IMF is separately spliced ​​with the preprocessed meteorological data to form multiple test data and normalized. Each test data is paired with a trained improved Transformer network. The normalized multiple test data are respectively input into their respective improved Transformer networks for prediction to obtain the corresponding prediction results. Finally, all the prediction results are superimposed and reconstructed as the final prediction results for output.

2. The natural gas load forecasting method based on the improved Transformer network according to claim 1 is characterized in that: In step 2), the collected data is preprocessed, including: Identify and process missing values ​​in the data and use the same-period filling method to supplement them; Detect and delete outliers, and visually observe and discover data points that deviate from the overall trend by drawing scatter plots and line graphs.

3. The natural gas load forecasting method based on the improved Transformer network according to claim 2 is characterized in that: In step 3), the preprocessed natural gas load data is decomposed using the variational mode decomposition method VMD. The specific process is as follows: First, initialize the VMD parameters, including the number of modes K, the center frequency of each mode ω k , modal function u k , penalty parameter α, tolerance ε and Lagrange multiplier λ; Secondly, iteratively update the modal function u k , center frequency ω k and the Lagrange multiplier λ: In the formula, represents the modal function u at the n+1th iteration k , represents the sum of all modes except the kth mode at the n+1th iteration, where i is the index and ω represents the frequency, represents the center frequency of the kth mode function at the nth iteration, represents the center frequency of the kth mode function at the n+1th iteration, represents the time-weighted energy integral of the kth mode function, represents the total energy integral of the kth mode function, λ n represents the Lagrange multiplier at the nth iteration, λ n+1 represents the Lagrange multiplier at the n+1th iteration, τ represents the step size parameter used to adjust the update speed, f is the natural gas load sequence, represents the sum of all modal functions; Next, determine whether the convergence condition is met: if the update amount of the modal function is less than the set tolerance ε, stop the iteration; otherwise, return to continue the iteration, as shown below: In the formula, represents the modal function u at the nth iteration k , represents the bi-norm, ε is the preset tolerance used to judge convergence; Finally, the envelope entropy of each modal function is calculated, and all envelope entropies are added to obtain the total envelope entropy. The total envelope entropy under different K values ​​is compared, and the K value corresponding to the minimum total envelope entropy is selected as the final decomposition mode number, which is expressed as follows: In the formula, E k represents the envelope entropy of the kth IMF, A k (m) represents the instantaneous envelope of the kth IMF at the mth moment, M represents the length of the natural gas load sequence, that is, the number of data points contained in the IMF, and E total represents the total envelope entropy of all IMFs.

4. The natural gas load forecasting method based on the improved Transformer network according to claim 3 is characterized in that: In step 4), the IMF and the preprocessed meteorological data are combined in the order of the time series to form a test data. The test data is normalized using MinMaxScaler to scale the data to the interval [0,1] to eliminate the dimensional differences between the data. The normalization formula is as follows: In the formula, x is the original data, x' is the normalized data, and x min is the minimum value in the data to be tested, x max is the maximum value in the data to be tested.

5. The natural gas load forecasting method based on the improved Transformer network according to claim 4 is characterized in that: In step 4), the normalized test data is input into the improved Transformer network. First, the position information is embedded through the position encoding formula. The calculation formula is as follows: In the formula, pos represents the position, index represents the dimension index, and d model Indicates dimension, PE (pos,2index) Represents the sine value of the 2nd index dimension at position pos, PE (pos,2index+1) Represents the cosine value of the 2ndindex+1th dimension at position pos; The time series with embedded position information is input into the encoder, which contains a multi-head attention mechanism Multi-Head Attention and a feed-forward neural network layer to capture global dependencies and local patterns in the sequence. Multi-Head Attention plays a key role in achieving accurate prediction. Each attention head can capture information in different representation subspaces by learning different weights. The outputs of multiple attention heads are then linearly transformed and connected together. The calculation formula is as follows: MultiHead=Concat(head1,head2,...,head j ,…,head n′ )*W o In the formula, MultiHead represents multi-head attention, Concat(·) represents connecting the outputs of each head, and head n′ represents the n′th attention head, W o represents the output weight matrix; In Multi-Head Attention, each attention head performs the calculation process of the self-attention mechanism, thereby focusing on the internal information of the sequence. The calculation formula is as follows: In the formula, Query represents the input of the current position, which is used to calculate the attention weight of this position and other positions; Key represents the input of other positions, which is used to match the query and calculate the attention weight; Value represents the input of other positions, which is used to generate the output of the current position according to the weighted result of the attention weight; denote the linear transformation matrices for query, key, and value, respectively; Attention(·) denotes the computation of the self-attention mechanism; Next, the extracted feature information is fed into the decoder to generate a preliminary prediction result; wherein the decoder is composed of an LSTM network and a fully connected layer, as follows: The LSTM network uses three key mechanisms, namely the forget gate, the input gate and the output gate, to regulate the flow of information, thereby selectively retaining, discarding and updating information; The forget gate will be based on the current input x t and the hidden state h at the previous moment t-1 And the corresponding weight matrix W f and the bias term b f To calculate a value f between 0 and 1 t , represents the amount of information that needs to be retained, and is converted into the following formula: f t =sigmoid(W f .[h t-1 ,x t ]+b f ) The input gate first determines which new information should be updated or written through the sigmoid activation function, and then creates a candidate vector through the tanh activation function To update the cell state; the input gate will be based on the current input x t and the hidden state h at the previous moment t-1 And the corresponding weight matrix W i , W c and the bias term b i 、b c Calculate a value i between 0 and 1 t , represents the degree to which the input gate is to be updated, and a candidate vector The converted formula is as follows: In the formula, C t-1 Indicates the cell state at the last moment, C t Indicates the cell state at the current moment; The output gate first determines the cell state C through the sigmoid activation function t Which parts of the cell state will be passed out, and then the value of the cell state is converted to between -1 and 1 through the tanh activation function, and finally the output of the tanh activation is multiplied element by element with the output of the output gate to generate the final output value; the output gate will be based on the current input x t and the hidden state h at the previous moment t-1 And the corresponding weight matrix W o and the bias term b o Calculate a value O between 0 and 1 t , represents the amount of information to be transmitted by the output gate, which is converted into the following formula: In the formula, h t Indicates the hidden state at the current moment; And a residual connection mechanism is introduced between the encoder and the LSTM network. The input is directly added to the output through the residual connection mechanism. The calculation formula is as follows: output=dropout(LSTM(x input ))+x input In the formula, output represents the preliminary prediction result, and dropout(·) randomly sets the output of some neurons to zero, thereby reducing the network's dependence on specific neurons and preventing the network from overfitting. input Represents the features extracted by the encoder; The output is input into the fully connected layer for re-prediction, and the result is used as the prediction result of the improved Transformer network; in the fully connected layer, each neuron is connected to all neurons in the previous layer, and an activation function ReLU is connected after each fully connected layer; Finally, the prediction results of each improved Transformer network are superimposed to form the final prediction result.

Citation Information

Cited By

  • CNN-Transform-based short-term wind power prediction method

    CN119692392A

  • Abrasion prediction method and system for suspension clamp

    CN120850665A

  • A method and system for predicting wear of a suspension clamp

    CN120850665B

  • Natural gas load prediction method and device

    CN122332828A