Load prediction method based on decomposition representation and attention encoder network

Abstract

The invention relates to a load prediction method TSR-MAEN based on decomposition representation and an attention encoder network. Firstly, the algorithm provides a trend-seasonal decomposition representation method based on comparative learning so as to enhance the decoupling performance of a model on a trend component and a seasonal component of a load sequence and the representation capability of decomposition characteristics. Secondly, in order to deeply mine and decompose time sequence information contained in a feature hidden space and dependency between features, the algorithm proposes a mixed attention encoder network; through segmented embedding, a trend-season dynamic projection mixed self-attention mechanism and a multi-head mixed output mechanism, trend and season multi-dimensional features after load sequence decomposition are effectively utilized to express and enhance the long-range dependence information extraction capability of the model. Experiments show that the TSR-MAEN algorithm provided by the invention has excellent performance in an electrical load prediction task of oil and gas development, and the prediction error is reduced by 8.51% compared with that of an optimal baseline algorithm.

Description

Technical Field

[0001] This invention relates to the field of industrial time-series data analysis, specifically a load prediction algorithm based on trend-seasonal decomposition representation and a hybrid attention encoder network. Background Technology

[0002] The electrical load of oil and gas development refers to time-series data obtained by sampling the instantaneous power of oil and gas development equipment at certain time intervals. During oil and gas field development, the electrical load along a development line is typically sampled uniformly, and the sampled data is reported and statistically analyzed via an IoT gateway. Predicting the electrical load of oil and gas equipment along the line is crucial for the oil and gas development process. Accurate load prediction allows for optimized energy management, reducing energy consumption and operating costs. Furthermore, load prediction data provides a foundation for intelligent management and decision support. Combining big data analytics and artificial intelligence technologies enables comprehensive monitoring and intelligent optimization of the oil and gas field production process, improving overall management levels and the scientific basis of decision-making. Therefore, how to achieve accurate prediction of the electrical load of oil and gas equipment along the line is a problem worthy of research.

[0003] Due to the complex characteristics of electrical equipment used in oil and gas exploration and development, load time series often contain complex compositional patterns such as high-frequency components, seasonal components, and trend components. However, end-to-end prediction models often directly model the load time series, failing to effectively model and analyze these component characteristics. To improve the modeling capability of time series, researchers have proposed some feature engineering methods. By decomposing various components in the time series or mapping the time series to a latent space, the prediction model can better understand the structure and variation patterns of the time series, thereby improving the model's prediction performance. For example, Lingling et al. decomposed the input time series into multiple modal components with different center frequencies and finite bandwidths using variational mode decomposition, and processed each component separately to reduce prediction errors. However, this method is difficult to determine the number of decomposition modes when there is no prior information or a lack of understanding of the time series composition. Setting it too high can lead to decomposition redundancy, while setting it too low can cause information loss. MinFan et al. introduced a trend-seasonal decomposition method based on STL, extracting trend and seasonal components separately through locally weighted multinomial regression, and updating the residuals iteratively. This method only considers a single seasonal cycle and has significant computational limitations for large-scale data. Zhihan et al. introduced representation learning to map time series data to a multidimensional latent space and decoupled the temporal components through contrastive learning and variational inference. The latent space possesses stronger information representation capabilities, enabling the model to effectively encode complex patterns in the time series, thereby improving the ability to represent sequence patterns. Representation learning can map the original sequence data to a multidimensional feature latent space. This process not only preserves the inherent structure of the data but also enhances the semantic information of the sequence under different periodicity and step size patterns. Furthermore, in the high-dimensional feature latent space, not only does each feature channel encode the trend or periodic information of the sequence, but there are also often dependencies between the feature channels. However, most current load time series prediction algorithms based on decomposed representations directly decode the representation features into prediction results using algorithms such as regression, without deeply exploring the temporal information contained in the feature latent space and the dependencies between features.

[0004] In summary, existing load forecasting models still have certain limitations. End-to-end forecasting models fail to effectively decompose and utilize the complex features in load time series, while models based on decomposition representations, although extracting features from load time series, fail to deeply explore the time series information and dependencies between features in the latent feature space. Therefore, both suffer from insufficient forecasting accuracy in complex fluctuating load scenarios involving electricity consumption by oil and gas equipment. Summary of the Invention

[0005] The purpose of this invention is to propose a load prediction algorithm based on trend-seasonal decomposition representation and a hybrid attention encoder network, which enhances the model's ability to decouple and represent load components while fully extracting the temporal patterns and dependency information contained in the decomposition features, thereby improving the prediction effect for complex load time series.

[0006] The objective of this invention is achieved as follows:

[0007] First, a trend-seasonal decomposition representation method based on contrastive learning is proposed to enhance the model's decoupling performance of the trend and seasonal components of the load sequence and its ability to represent decomposed features. Second, to deeply explore the temporal information and inter-feature dependencies contained in the latent space of the decomposed features, a hybrid attention encoder network is proposed. Through segmented embedding, trend-seasonal dynamic projection cross-self-attention mechanism, and multi-head hybrid output mechanism, this network effectively utilizes the multi-dimensional trend and seasonal feature representations after load sequence decomposition and enhances the model's ability to extract long-range dependency information.

[0008] The specific method is as follows:

[0009] A load prediction algorithm based on trend-seasonal decomposition representation and a hybrid attention encoder network includes the following steps:

[0010] Step 1: Trend-Season Decomposition. Input the load time series data into the backbone encoder, trend decomposer, and seasonal decomposer to extract the trend and seasonal components of the sequence and convert them into trend multidimensional features and seasonal multidimensional features.

[0011] Step 2, Representation Embedding: The multidimensional feature sequence is sliced, encoded, and embedded by channel to transform it into an embedding representation, which reduces the size of the original input feature data while introducing sequence-level semantic information.

[0012] Step 3: Calculate the trend-seasonal dynamic projection hybrid attention. Calculate the time attention, trend feature channel dynamic projection attention, and seasonal feature channel dynamic projection attention separately. Extract the dependency information within the time series and between channels. Furthermore, reduce the feature dimension of the calculation through the dynamic projection mechanism to optimize the attention calculation speed.

[0013] Step 4: Multi-head hybrid output. The multi-head output results of each attention are calculated by merging windows to improve the ability to extract multi-scale information from the sequence.

[0014] Step 5: Output the results. The sequence information extracted by the encoder is aggregated through the linear output layer, and the sequence information is mapped to the predicted time series values ​​through a fully connected network, and the prediction results are output.

[0015] The positive effects of this invention are:

[0016] To effectively decompose and represent the trend and seasonal components of load time series and fully extract the temporal patterns and dependency information contained in the decomposed features to improve prediction performance, this invention proposes a load prediction algorithm, TSR-MAEN, based on trend-seasonal decomposition representation and a hybrid attention encoder network. First, a trend-seasonal decomposition representation method based on contrastive learning is proposed to enhance the model's decoupling performance of the trend and seasonal components of the load sequence and its ability to represent decomposed features. Second, to deeply mine the temporal information and inter-feature dependencies contained in the latent space of the decomposed features, a hybrid attention encoder network is proposed. This network effectively utilizes the multi-dimensional trend and seasonal feature representations after load sequence decomposition through segmented embedding, a trend-seasonal dynamic projection hybrid self-attention mechanism, and a multi-head hybrid output mechanism, enhancing the model's ability to extract long-range dependency information. Experiments show that this algorithm achieves excellent performance in predicting electricity load for oil and gas development, reducing the prediction error by 8.51% compared to the best baseline algorithm. Attached Figure Description

[0017] Figure 1 This is a framework diagram of a trend-seasonal decomposition representation method based on contrastive learning.

[0018] Figure 2 This is a diagram of a hybrid attention network structure.

[0019] Figure 3 This is a diagram of a trend-seasonal dynamic projection hybrid attention mechanism.

[0020] Figure 4 It is a decomposed representation of the instance group diagram.

[0021] Figure 5 This is a comparison chart of prediction results. Detailed Implementation

[0022] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0023] Step 1: Trend-Seasonal Decomposition. For example... Figure 1 This presents the overall framework of the trend-seasonal decomposition representation algorithm based on contrastive learning proposed in this invention. It mainly includes a backbone encoder, a trend decomposer, and a seasonal decomposer, and utilizes contrastive learning to learn the optimal seasonal-trend decoupling representation. The decomposition process is summarized as follows: First, the backbone encoder processes the input load sequence X = [x1, x2, ..., x...] of length T. T Mapping to a high-dimensional feature space d yields the intermediate representation of the sequence. Input M into the trend decomposer and the seasonal decomposer respectively, and calculate the characteristic representations of the trend and the season respectively. Where D TThe superscript T represents the trend, distinguishing it from the sequence length T. Finally, the combined outputs yield the feature representation after the sequence trend-seasonal decomposition.

[0024] The trend decomposer is based on an autoregressive model and consists of L+1 autoregressive experts. Each expert is implemented with d input channels and d... t A one-dimensional causal convolutional neural network with output channels, where the kernel size of the i-th expert is 2. i Each expert receives an intermediate representation M as input and outputs a matrix. The causal convolution output of the i-th expert is shown below:

[0025]

[0026] Average pooling is performed on the outputs of all autoregressive experts to obtain the trend representation D of the sequence. T As shown below:

[0027]

[0028] in This indicates the number of autoregressive experts.

[0029] The trend decomposer uses temporal contrastive loss to learn discriminative trend representations. The calculation process of temporal contrastive loss is summarized as follows: First, for each input sequence, two views are generated through data augmentation and input into the encoder and momentum encoder respectively, obtaining the query vector and the corresponding positive sample key. A dynamic queue constructed using the momentum encoder caches the representations of historical samples and uses them as negative samples for the current sample. Next, the similarity with positive and negative samples is measured using dot product calculation and normalized using Softmax to calculate the probability of positive sample pairs. The positive sample score is divided by the sum of all sample scores and the negative logarithm is taken to obtain the temporal contrastive loss, as shown below:

[0030]

[0031] Where N is the total number of samples, q i Let k be the query vector for the i-th sample. i Let k be the positive sample key of the i-th sample. j The negative sample key is the negative sample sampled from the negative sample queue, where K is the total number of negative samples and T is the temperature coefficient.

[0032] The seasonal decomposer extracts seasonal feature representations based on frequency domain analysis. First, the intermediate representation M is mapped from the time domain to the frequency domain using a Discrete Fourier Transform (DFT). A DFT is then performed on each dimension of the intermediate representation M to obtain the frequency domain matrix, with the number of frequencies being... Then, a frequency domain learning layer is introduced, which compresses the d-dimensional frequency domain to d-dimensional space through complex linear transformation. s This allows for interaction between different dimensions in the frequency domain. Finally, the inverse Fourier transform is used to transform the frequency domain representation back to the time domain, yielding the seasonal features of the sequence. The entire calculation process is as follows:

[0033]

[0034] Similar to trend decomposers, seasonal decomposers learn parameters through momentum contrastive learning. Since data augmentation can be interpreted as intervention in error variables, the seasonality of the sequence remains unchanged. A frequency domain contrastive loss function is introduced to distinguish different periodic patterns at a specified frequency. The input to the loss function is the frequency domain representation before the inverse Fourier transform. Their values ​​are all complex Fourier coefficients, which can be uniquely represented by the amplitude |Fi,:| and the phase φ(Fi,:). The loss function is calculated from both the amplitude and phase components, as shown below:

[0035]

[0036] Finally, the overall loss function of the trend decomposer and the seasonal decomposer is expressed as follows:

[0037]

[0038] Where 'a' is a balancing factor, used to weigh the weights of trend factors and seasonal factors during decomposition.

[0039] Step 2: Represent the embedding. For example... Figure 2 As shown, the embedding layer represents the input to the hybrid encoder network. It is responsible for segmenting, encoding, and embedding the input trend-seasonal features into a representation matrix, reducing the size of the original input feature data and introducing sequence-level semantic information into the representation matrix. To reduce the computation time of temporal attention and cross-channel attention mechanisms and enrich the semantic information carried by each representation vector in the representation matrix, this step employs the idea of ​​patching. The sequence is divided into multiple subsequences, and then each subsequence is linearly mapped and embedded to generate a representation vector for each subsequence. Compared to point-to-point generation of representation vectors, this method encodes the sequence semantic information into the representation vector through subsequence mapping, and reduces the number of representation vectors from the original sequence length to the number of subsequences, thus reducing the computation time for subsequent attention. The network input is the high-dimensional features after trend-seasonal decomposition. For each feature channel, perform subsequence segmentation, that is, take N subsequences of length P with a step size of S, and transform the input into a subsequence matrix. Then the subsequence matrix D pEach subsequence segment in the MLP is mapped to the latent feature space d through the MLP layer F. H Since the original sequence does not contain the position information of each point in the sequence, the position embedding PE based on sine and cosine functions proposed in Transformer is introduced. The PE is added to the latent feature space to form the representation matrix X. The whole process is as follows:

[0040] X = [F(Dp) + PE]

[0041] Where, represents a matrix Number of segments

[0042] Step 3: Calculate the trend-seasonal dynamic projection hybrid attention. To extract the time dimension and inter-channel temporal dependency information from the high-dimensional features after trend-seasonal decomposition, a trend-seasonal dynamic projection attention mechanism is introduced, such as... Figure 2 The encoder, by integrating trend-seasonal decomposition and cross-channel attention, not only improves the ability to extract dependencies between feature channels but also effectively reduces the time complexity of attention computation through dynamic projection. Furthermore, it horizontally segments the two independently distributed high-dimensional features of trend and season and performs attention computation separately, avoiding erroneous correlations between unrelated channels while reducing the scale of attention computation. The specific calculation process of trend-seasonal dynamic projection attention is as follows:

[0043] (1) Construct the weight matrix for the multi-head self-attention mechanism. First, obtain the input query matrix, key matrix, and value matrix through linear projection, as shown below:

[0044] Q = XW q K = XW k V = XW v

[0045] in, W q W k W v Linear projection layers for query, key, and value, respectively.

[0046] Then, Q, K, and V are transformed into H sets of multi-head weight matrices through linear projection. { Q i},{K i},{V i},in d head For head attention dimension.

[0047] (2) Calculate temporal attention. Based on the multi-head self-attention mechanism, for each feature dimension, the attention information between tokens is calculated according to the time dimension to obtain the sequence dependency relationship in the time dimension. First, the attention weight matrix is ​​sliced ​​according to the dimension, that is, for each set of weight matrix Q i ,K i V i Tokens divided into d channels in

[0048] Then, self-attention computation for multiple heads is performed in parallel for each feature channel. Unlike the standard multi-head concatenation and projection output method, the representation vectors of each head are merged using multi-head hybrid output to obtain the temporal attention computation result for a single feature channel, as shown below:

[0049]

[0050] Finally, the time attention calculation results for each channel are concatenated line by line to obtain the time attention output, as shown below:

[0051]

[0052] in,

[0053] (3) Calculate cross-channel attention for trend features. Based on the multi-head self-attention mechanism, the multidimensional embedding of trend features is treated as a multi-channel variable. Attention information is calculated according to the channel dimension to obtain the dependency relationship between multiple variables of the trend features. First, the attention weights are sliced ​​according to time, that is, for each set of weight matrices Q i ,K i V i Divide the sequence according to its position, and identify the trend characteristic channels at the same position, i.e., the first d. t Each channel token is grouped into the same group, resulting in N weight matrix slices. in j = 1, 2, 3, ..., N. After segmentation, dynamic projection is used to transform d. t Each channel is mapped to l t There are 1 channel, and there is 1 t <<d t The channel information is compressed and summarized. Specifically, dynamic projection maps the weight matrix through the MLP layer. The channel dimension is from d t Reduce to l t First, the weight matrix is ​​input into the MLP layer. and Then, softmax is used for normalization to calculate the projection matrix. Then, the product of the projection matrix and the weight matrix is ​​calculated to obtain the dimensionality-reduced weight matrix. As shown below:

[0054]

[0055] Wherein, projection matrix Dimensionally reduced weight matrix

[0056] Then, similar to temporal attention, multi-head self-attention computation is performed on the input dimensionality-reduced weight matrix. The representation vectors of each head are merged using the multi-head hybrid output to obtain the cross-channel attention computation result for trend features at a single time point.

[0057] (4) Similarly, calculate the cross-channel attention for seasonal features. First, slice the attention weights by time, that is, for each set of weight matrices Q i ,K i V i Segment by sequence position, and separate the seasonal feature channels at the same position, i.e., the latter d s Each channel token is grouped into the same group, resulting in N weight matrix slices. in After segmentation, dynamic projection is used to segment d. s Each channel is mapped to l s There are 1 channel, and there is 1 s <<d t The channel information is compressed and summarized. Specifically, dynamic projection maps the weight matrix through the MLP layer. The channel dimension is from d t Reduce to l s First, the weight matrix is ​​input into the MLP layer. and Then, softmax is used for normalization to calculate the projection matrix. Then, the product of the projection matrix and the weight matrix is ​​calculated to obtain the dimensionality-reduced weight matrix. Multi-head self-attention computation is performed on the input dimensionality-reduced weight matrix. The representation vectors of each head are merged using the multi-head hybrid output to obtain the cross-channel attention computation result of seasonal features at a single time point.

[0058] Step 4: Multi-head merging output. The next-stage representation vector is generated by merging adjacent position representation vectors within the same head and between different heads, replacing the standard multi-head merging method and extracting multi-scale information from the same head. Assume the multi-head attention mechanism contains H heads, each outputting N representation vectors, with each representation vector having a latent space size of d. The set of all output representation vectors can be represented as a 2D matrix. in Let h = 1, 2, ..., H; n = 1, 2, ..., N represent the nth output of the h-th column. The sliding window slides along each column with a step size of 1, and the window size is H × (2k + 1), where k is an adjustable size of the adjacent region of interest. For the first k columns and the last k columns, the window is filled by repeating the first or last column. The sliding window is represented as follows:

[0059] w n ={O(x,y)∣1≤x≤H,nk≤y≤n+k}

[0060] Where n∈{1,2,…,N} is the number of merged tokens.

[0061] Step 5: Output Results. The sequence information extracted by the linear output layer is aggregated and mapped to the predicted time series values ​​through a fully connected network, and the prediction results are output.

[0062] Figure 4 This is a breakdown of the experimental group diagrams. Among them, Figure 4 (a) is the original load sequence before decomposition representation. Figure 4 (b) The trend-seasonal multidimensional features after decomposition of the original load series are shown. The trend-seasonal representation of the original load series is captured into six subsequences, R1-R6. R1, R2, and R3 represent the seasonal decomposition results, all exhibiting strong periodicity patterns; R4, R5, and R6 represent the trend decomposition results, clearly reflecting the trend changes of the original series. To more intuitively demonstrate the differences in trend-seasonal features after decomposition of the original series, the trend-seasonal representation is transformed into... Figure 4 (c) shows the t-SNE plot. Each point in the plot represents a time step, with colors indicating the time intervals within that step. The position of each point in two-dimensional space represents the distribution corresponding to its seasonal trend characteristics. The data point distribution in the plot demonstrates that the high-dimensional representation obtained from the decomposition can effectively distinguish different trends and seasonal distributions, with each category focusing on a specific time pattern.

[0063] Figure 5 This is an example of the prediction performance compared to a baseline model. As can be seen from the figure, the TSR-MEAN algorithm proposed in this invention predicts load trends and periodic changes more accurately than other baseline models, achieving more precise load forecasting.

Claims

1. A load prediction method based on decomposition representation and attention encoder network, characterized in that, The method includes the following steps: Step 1: Trend-Season Decomposition. Input the load time series data into the backbone encoder, trend decomposer, and seasonal decomposer to extract the trend and seasonal components of the sequence and convert them into trend multidimensional features and seasonal multidimensional features. Step 2, Representation Embedding: The multidimensional feature sequence is sliced, encoded, and embedded by channel to transform it into an embedding representation, which reduces the size of the original input feature data while introducing sequence-level semantic information; Step 3: Calculate the trend-seasonal dynamic projection hybrid attention. Calculate the time attention, trend feature channel dynamic projection attention, and seasonal feature channel dynamic projection attention respectively. Extract the dependency information within the time series and between channels. Furthermore, reduce the feature dimension of the calculation through the dynamic projection mechanism to optimize the attention calculation speed. Step 4: Multi-head hybrid output. The multi-head output results of each attention are calculated by merging windows to improve the ability to extract multi-scale information from the sequence. Step 5: Output the results. The sequence information extracted by the encoder is aggregated through the linear output layer, and the sequence information is mapped to the predicted time series values ​​through a fully connected network, and the prediction results are output.

2. The load prediction method based on decomposition representation and attention encoder network as described in claim 1, characterized in that, The load prediction algorithm in step 1, based on decomposition representation and attention encoder network, utilizes a trend decomposer based on an autoregressive model to extract sequence trend features. It consists of L+1 autoregressive experts, each implemented with d input channels and d... t The output channel is a one-dimensional causal convolutional neural network, and finally, the outputs of all autoregressive experts are averaged and pooled to obtain the trend representation D of the sequence. T The calculation process is as follows: The kernel size of the i-th expert is 2. i Each expert receives an intermediate representation M as input and outputs a matrix. The trend decomposer uses temporal contrastive loss to learn a discriminative trend representation. The formula for calculating temporal contrastive loss is as follows: where N is the total number of samples, q i is the query vector of the i-th sample, k i is the positive sample key of the i-th sample, k j is the negative sample key sampled from the negative sample queue, K is the total number of negative samples, and T is the temperature coefficient.

3. The load prediction method based on decomposition representation and attention encoder network as described in claim 1, characterized in that, The process of the trend-seasonal decomposition representation algorithm based on contrastive learning in step 1 involves extracting seasonal feature representations using a seasonal decomposer based on frequency domain analysis, and compressing the frequency domain of d dimensions to d using a learnable Fourier layer. s Each dimension enables interaction between different frequency domains, and the calculation formula is as follows: The seasonal decomposer updates its parameters using frequency domain contrast loss, calculating both amplitude and phase, as shown in the following formula; 4. The load prediction method based on decomposition representation and attention encoder network as described in claim 1, characterized in that, The representation embedding layer of the hybrid attention network in step 2 is characterized by dividing the sequence into multiple subsequences, then performing linear mapping and position embedding on each subsequence to generate a representation vector for each subsequence, thereby reducing the size of the original input feature data and introducing sequence-level semantic information into the representation matrix.

5. The load prediction method based on decomposition representation and attention encoder network as described in claim 1, characterized in that, The attention calculation process of the hybrid attention network in step 3 is characterized by performing attention calculations along both time and channel dimensions, enabling the model to extract dependency information between the time axis and channels; introducing a dynamic projection mechanism to compress the number of channels by identifying key feature dimensions and dynamically optimizing feature weights; furthermore, horizontally segmenting the two independently distributed high-dimensional features of trend and season, and calculating channel attention separately to avoid erroneous correlations between unrelated channels, thus reducing the scale of attention calculations. Taking cross-channel attention of trend features as an example, dynamic projection maps the weight matrix through an MLP layer. The channel dimension is from d t Reduce to l t First, the weight matrix is ​​input into the MLP layer. and Then, Softmax is used for normalization to calculate the projection matrix. Then, the product of the projection matrix and the weight matrix is ​​calculated to obtain the dimensionality-reduced weight matrix. The calculation formula is as follows: Wherein, projection matrix Dimensionally reduced weight matrix 6. The load prediction method based on decomposition representation and attention encoder network as described in claim 1, characterized in that, The multi-head hybrid output process in step 4 is characterized by generating the representation vector for the next stage by merging the representation vectors of adjacent positions within the same head and between different heads, extracting multi-scale information from the same head, and using the following formula for the sliding window during merging: w n ={O(x,y)∣1≤x≤H,n-k≤y≤n+k} Where n∈{1,2,...,N} is the number of merged tokens, H is the number of multi-head tokens, N is the number of representation vectors contained in each head, and d is the size of the latent space of the representation vectors. To represent a set of vectors, This represents the nth output from the h-th head.

Images