Non-stationary time series prediction method and system based on hierarchical wavelet attention

Abstract

The application discloses a non-stationary time series prediction method and system based on hierarchical wavelet attention, and proposes a decomposition-independent modeling-adaptive fusion refined prediction paradigm; first, through wavelet derivative transformation, the dynamic characteristics of endogenous time series data are effectively separated into multiple frequency scales, and non-stationary components such as mutations are strengthened; for each frequency scale, a separate time series encoder is used for hierarchical independent modeling; then, through an adaptive scale fusion mechanism, the encoded features of all frequency scales are dynamically weighted and fused at each time point, so that the model can focus on the most critical dynamic information at the moment. This method fundamentally solves the modeling problem of complex dynamics in non-stationary time series, and significantly improves the accuracy and robustness of prediction.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to a non-stationary time series prediction method and system based on hierarchical wavelet attention. BACKGROUND

[0002] Time series prediction plays a crucial role in many fields such as power load, weather, financial transactions, and traffic flow. Deep learning models based on the Transformer architecture have achieved remarkable success in capturing long-term dependencies of time series. However, existing models still face significant challenges when dealing with non-stationary time series commonly seen in the real world.

[0003] Real-world time series data, such as power consumption data, often contains abrupt points caused by extreme weather, policy changes, or equipment failures, as well as transient oscillations caused by short-term external factors such as holiday effects and market fluctuations. These non-stationary characteristics are manifested as high-frequency transient components in the signal.

[0004] Traditional Fourier transform-based analysis methods can identify the frequency components of the signal, but they lose the time dimension information and cannot locate the time points of the abrupt changes. Existing mainstream time series prediction models usually treat the time series as a whole or segment them into time slices for processing, trying to learn multiple patterns such as trends, periodicities, abrupt changes, and oscillations with a single model structure. This approach can easily lead to interference between different types of signal patterns during learning. For example, the model may smooth out critical abrupt information to adapt to stable periodicity; conversely, it may overfit in stable areas to capture abrupt changes.

[0005] Specifically, when applying deep learning models based on the Transformer architecture to electricity consumption prediction tasks, it is found that the model's prediction performance is poor in the areas of abrupt points and short-term oscillations. The root cause lies in the model's inability to effectively decouple and model the stationary components (such as seasonal trends) and non-stationary components (such as abrupt changes) of the data.

[0006] Therefore, there is an urgent need for a new technical solution that can effectively distinguish and learn stationary and non-stationary dynamics in time series separately, thereby improving the prediction accuracy of real-world time series containing complex dynamics. SUMMARY

[0007] The present application aims to solve the problem of insufficient modeling of non-stationary characteristics (especially abrupt points and transient oscillations) by existing time series prediction models, as well as the mutual interference of different dynamic patterns in modeling. It provides a non-stationary time series prediction method and system based on hierarchical wavelet attention, which can significantly improve the prediction accuracy of non-stationary time series.

[0008] To achieve the above object, the core idea of the present application is to propose a new paradigm of hierarchical independent modeling and adaptive fusion of non-stationary dynamics. Instead of simply parallel processing time domain and frequency domain, this method goes deep into the wavelet domain, and makes fine and decoupled time series dependent modeling of different dynamic characteristics represented by different frequency scales. The present application is implemented through the following technical scheme. A non-stationary time series prediction method based on hierarchical wavelet attention, comprising the following steps:

[0009] a. receiving endogenous time series data;

[0010] b. extracting features from the endogenous time series data to obtain an endogenous feature representation at each time slice position, which includes:

[0011] i. wavelet derivative transformation: differentiating the endogenous time series data, and then performing multi-level discrete wavelet transform on the differentiated endogenous time series data to obtain a series of detail coefficient sequences belonging to multiple different frequency scales;

[0012] ii. hierarchical independent modeling: for each frequency scale, the corresponding detail coefficient sequence is taken as an independent time series and input into a time series encoder for time series dependent modeling to learn the dynamic pattern within the frequency scale, and an encoded feature sequence containing context information at the frequency scale is obtained;

[0013] iii. adaptive scale fusion: for each time slice position, a scale fusion attention mechanism is used to dynamically weight and fuse the encoded features from all frequency scales at the time slice position to generate a wavelet domain feature representation of the time slice position that integrates multi-scale non-stationary information, as part of the endogenous feature representation;

[0014] c. generating a prediction value for the future time series based on the endogenous feature representation.

[0015] Further preferably, in step b, the feature extraction process further includes: processing the endogenous time series data through a time series encoder to obtain a time domain feature representation; and the wavelet domain feature representation generated in step iii and the time domain feature representation are fused through a gated fusion mechanism to jointly constitute the endogenous feature representation.

[0016] Further preferably, in step ii, the time series encoder used for time series dependent modeling of the detail coefficient sequences of different frequency scales is weight-shared.

[0017] Further preferably, the timing encoder is a Transformer architecture-based encoder comprising a multi-head self-attention mechanism.

[0018] Further preferably, the scale fusion attention mechanism in step iii is implemented by a self-attention network that calculates the correlation between different frequency scale features and outputs the weighted fused features.

[0019] Further preferably, the gating fusion mechanism dynamically calculates the fusion weight according to the input time domain feature representation and wavelet domain feature representation, and performs weighted summation on the time domain feature representation and wavelet domain feature representation.

[0020] Further preferably, it further comprises:

[0021] receiving exogenous variable data related to the endogenous time series data;

[0022] embedding the exogenous variable data into an exogenous feature representation;

[0023] Before generating the prediction value, the fused endogenous feature representation and the exogenous feature representation are interacted and integrated.

[0024] The application also provides a non-stationary time series prediction system based on hierarchical wavelet attention, comprising:

[0025] one or more processors;

[0026] a memory having computer executable instructions stored thereon;

[0027] When the instructions are executed by the one or more processors, the system can perform the above-mentioned non-stationary time series prediction method based on hierarchical wavelet attention.

[0028] The application has the following characteristics:

[0029] (1) Wavelet derivative transform and multi-scale dynamic separation:

[0030] First, the received historical endogenous time series is subjected to first-order difference processing. This operation is similar to differentiation on discrete signals, and its purpose is to amplify the "change rate" of the signal, thereby numerically strengthening the non-stationary features such as abrupt changes and oscillations. Subsequently, the sequence after difference is subjected to multi-level discrete wavelet transform. This step is defined as "wavelet derivative transform". Through this transformation, the dynamic characteristics of the original sequence are effectively decomposed into multiple different frequency scales, each scale being represented by a group of detail coefficient sequences, corresponding to different frequency wave information respectively.

[0031] (2) Hierarchical independent modeling and timing dependent deep learning:

[0032] This step is the core innovation of the invention, which is called "hierarchical wavelet attention" mechanism. Unlike the prior art, which mixes all frequency information for processing, the invention independently and deeply models the time series for each frequency scale obtained in the previous step. Specifically, for each frequency scale j, we reorganize all the detail coefficients of the time slices at this scale into a time series sequence that is unique to this scale. Then, we use a time series encoder (such as Transformer) to independently encode this sequence to fully learn and capture the dynamic patterns and time dependencies specific to this frequency scale. Through this "hierarchical independent modeling" approach, high-frequency sudden dynamics and low-frequency periodic dynamics are learned in separate channels, completely avoiding interference between different types of information.

[0033] (3) Adaptive scale fusion and feature integration:

[0034] After independent modeling of each frequency layer, the information needs to be effectively integrated. For each time slice position i, the invention uses an "adaptive scale fusion" mechanism. This mechanism collects the feature vectors after encoding at all frequency scales for the time slice, and dynamically calculates and assigns weights through a scale fusion attention network. This allows the model to automatically determine which frequency scale information should be emphasized based on the specific circumstances of the current time slice. For example, when the sequence is stable, the model can automatically increase the attention to low-frequency trend features; when the sequence fluctuates sharply, it can automatically increase the weight of high-frequency sudden features. Finally, a single, comprehensive wavelet domain feature representation is generated.

[0035] (4) Feature fusion and final prediction:

[0036] The generated wavelet domain feature representation can be selectively gated and fused with the time domain features (representing stable components) extracted through traditional methods to obtain the final endogenous feature representation. Combined with the exogenous variable information input externally, the prediction head generates predicted values for future time series.

[0037] Through the above method, the invention transforms the prediction problem of non-stationary time series from a rough modeling of mixed patterns to a detailed processing flow of decomposition-independent modeling-adaptive fusion, thereby fundamentally improving the model's ability to capture complex dynamics and prediction accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 Flowchart of the invention's non-stationary time series prediction method based on hierarchical wavelet attention. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below. Apparently, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the protection scope of the present application.

[0040] With reference to Figure 1 , the embodiment provides a non-stationary time series prediction method based on hierarchical wavelet attention, comprising:

[0041] Step 101: data receiving. The system receives historical power load data as endogenous time series data , L is the length of the endogenous time series data, represents the data domain of the endogenous time series data, and exogenous variable data such as weather and holidays can also be received , represents the sample number dimension of the exogenous variable data, represents the number of types of exogenous variables.

[0042] Step 102: wavelet derivative transformation. This step aims to separate and enhance the non-stationary dynamics in the endogenous time series data. First, the first-order difference sequence of the endogenous time series data is calculated , where amplifies the rate of change of the signal, is the data at time t, is the data at time t-1, is the data at time t in the first-order difference sequence. Then, the first-order difference sequence is also divided into time slices. For each time slice , a level discrete wavelet transform (for example, using a Daubechies wavelet basis) is applied to obtain the approximation coefficient and group of detail coefficients . In this embodiment, to force the model to focus on non-stationarity, only the detail coefficients are used. Each group of coefficients represents the dynamic information of a specific frequency scale.

[0043] Step 103: hierarchical independent modeling. This step is the technical core of the present application, which decouples and deeply models the time series of each frequency scale. For each scale , the corresponding detail coefficients of all time slices are collected, j is the scale index, and i is the time slice position index, to form a time series of coefficient set For each element in apply a linear embedding layer to map it to a dimensional token, resulting in independent token sequences , where the dimension of the token sequence is . , represents the data field of the token sequence. Use a shared-weight temporal Transformer encoder (including multi-head self-attention layers and feed-forward network layers) to process each token sequence independently, calculate its internal temporal dependencies , and obtain the encoded feature sequence , where TemporalEncoder represents the temporal Transformer encoder function.

[0044] Step 104: Adaptive scale fusion. This step aims to intelligently fuse information from different frequency scales. For each time slice position , extract the corresponding feature vector from groups of feature sequences to form a new sequence , where represents the feature sequence of scale at time slice position , and T represents transposition. Input this sequence containing vectors into a small self-attention network (scale fusion attention module) to calculate the correlation between vectors and output a weighted average fusion vector . Concatenate the fusion vectors of all positions to obtain the final wavelet domain feature representation , where MeanPool is the mean pooling operation and SelfAttention is the self-attention mechanism.

[0045] Step 105: Optional feature fusion and prediction. To capture the stationary components in the sequence, endogenous time series data is divided into non-overlapping time slices of length , where is the length of each time slice. Apply a linear projection layer to each time slice to map it to a dimensional vector, resulting in time slice token sequence . At the same time, initialize a learnable global token for subsequent interaction with exogenous variables.

[0046] ​For each time slice position, the time domain token and the wavelet domain feature are concatenated. The concatenated vector is input into a linear layer and passed through a Sigmoid activation function to calculate a gating score. Then the final fused feature is calculated. Finally, the fused endogenous feature and the exogenous variable information after embedding are sent to an information interaction module, and the prediction value of the future time step is output through a prediction head .

[0047] Step 106: model training. The entire model is trained end-to-end by minimizing the mean square error (MSE) or other loss function between the predicted value and the true value .

[0048] Another embodiment of the present application provides a non-stationary time series prediction system based on hierarchical wavelet attention, comprising:

[0049] one or more processors;

[0050] a memory having computer executable instructions stored thereon;

[0051] When the instructions are executed by the one or more processors, the system can execute the above-mentioned non-stationary time series prediction method based on hierarchical wavelet attention. The method of the present application can be implemented on a system composed of one or more computers. The system includes a processor, a memory and an input / output interface.

[0052] It should be noted that the above-mentioned is only a specific embodiment of the present application, and obviously the present application is not limited to the above-mentioned embodiment, and there are many similar changes. All the modifications directly derived or thought from the disclosure of the present application by those skilled in the art should belong to the protection scope of the present application.

[0053] The above is only a preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A non-stationary time series prediction method based on hierarchical wavelet attention, characterized in that, The method comprises the following steps: a. Receiving endogenous time series data; b. Feature extraction is performed on the endogenous time series data to obtain an endogenous feature representation at each time slice position, which comprises: i. Wavelet derivative transformation: the endogenous time series data is differentiated, and then the differentiated endogenous time series data is subjected to multi-level discrete wavelet transformation to obtain a series of detail coefficient sequences belonging to different frequency scales; ii. Hierarchical independent modeling: for each frequency scale, the corresponding detail coefficient sequence is taken as an independent time series and input into a time series encoder for time-dependent modeling to learn the dynamic pattern within the frequency scale, thereby obtaining an encoded feature sequence of the frequency scale containing context information; iii. Adaptive scale fusion: for each time slice position, a scale fusion attention mechanism is used to dynamically weight and fuse the encoded features from all frequency scales at the time slice position to generate a wavelet domain feature representation of the time slice position, which comprehensively integrates multi-scale non-stationary information, as part of the endogenous feature representation; In step b, the feature extraction process further comprises: processing the endogenous time series data through the time series encoder to obtain a time domain feature representation; the wavelet domain feature representation generated in step iii and the time domain feature representation are fused through a gating fusion mechanism to jointly constitute the endogenous feature representation; the gating fusion mechanism dynamically calculates the fusion weight according to the input time domain feature representation and wavelet domain feature representation, and performs weighted summation on the time domain feature representation and wavelet domain feature representation; c. Generating a predicted value of a future time series based on the endogenous feature representation.

2. The method of claim 1, wherein, In step ii, the time series encoder used for time-dependent modeling of the detail coefficient sequences of different frequency scales is weight-shared.

3. The method of claim 2, wherein, The time series encoder is a Transformer architecture-based encoder containing a multi-head self-attention mechanism.

4. The method of claim 1, wherein, The scale fusion attention mechanism in step iii is realized by a self-attention network that calculates the correlation between features of different frequency scales and outputs the weighted fused features.

5. The method of claim 1, wherein, Further comprising: Receiving exogenous variable data related to the endogenous time series data; Embedding the exogenous variable data into an exogenous feature representation; Before generating the predicted value, the fused endogenous feature representation and the exogenous feature representation are interacted and integrated.

6. A non-stationary time series prediction system based on hierarchical wavelet attention, characterized in that, Comprise: One or more processors; A memory having computer executable instructions stored thereon; Wherein, when the instructions are executed by the one or more processors, the system can perform the method of any one of claims 1 to 5.

Images