Method for predicting time sequence in multi-level scene
By deconstructing the aggregation time series in multi-level scenarios using the frequency decomposition module and adaptive attention mechanism in the AnaNET model, the problem of periodic pattern entanglement in the aggregation time series is solved, and the accuracy and efficiency of prediction are improved.
Patent Information
- Application Number
- CN202410871731.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-01
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art cannot effectively deal with periodic pattern entanglement in aggregated time series in time series prediction in multi-level scenarios, resulting in prediction accuracy and inefficiency, and conventional methods ignore the complexity of aggregated time series.
Using a multi-level scenario prediction method, the high and low frequency components in the aggregated time series are deconstructed through the frequency decomposition module and the adaptive attention mechanism, and combined with seasonal trend decomposition, the AnaNET model is designed to deal with the timing dependence in multi-layer scenarios. relation.
It improves the accuracy and efficiency of time series prediction, can effectively deconstruct periodic pattern entanglement in the aggregated time series, enhances the model's processing ability on time series dependence, and achieves more accurate prediction.
Smart Images

Figure CN118939956A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of Internet of Things, and in particular to a method for predicting time series in multi-level scenarios. Background Art
[0002] With the rapid development of IoT technology and the widespread deployment of sensors, we have entered an era of data explosion. Most of the data collected by these sensors is in the form of time series, recording the status and activities of various devices and sensors at different time points. Time series, or a collection of data with timestamps and arranged in chronological order, has become extremely important due to its unprecedented value in recording historical status or activities and being able to judge future trends. Whether it is smart energy, smart transportation, smart manufacturing or computing power network, time series data plays a vital role in these cutting-edge applications.
[0003] Long-term prediction is a major challenge in time series prediction. Unlike short-term prediction, it requires the model to accurately understand recent trends and patterns while also being able to predict long-term changes. The difficulty of this type of prediction lies in the need to capture multiple periodic patterns and time series dependencies over a long period of time, while also taking into account possible emergencies and unknown factors that may occur over a long period of time.
[0004] In addition, multi-layer architecture is widely used in real-world applications, such as power, network, and transportation systems, with the support of the Internet. This multi-layer architecture brings unique challenges and opportunities to time series prediction. This architecture ensures that the data collected from individual devices and systems at all levels, referred to as Atomic Time Series (AtoTS) in this invention, can be effectively converged upward to form a more comprehensive and comprehensive top-level data view. However, this bottom-up data aggregation also means that the aggregated time series (AggTS) data formed by the upper layer contains the superposition and interweaving of many underlying periodic patterns, which brings complexity and chaos to time series prediction.
[0005] The aggregation of time series in multi-level scenarios can be regarded as aggregation behavior in two dimensions: time dimension and space dimension.
[0006] Time aggregation: In the process of forming aggregated time series data, different sampling frequencies have a significant impact on the temporal dependence of the data set, which in turn affects the characteristics of the observed periodic patterns. For example, consider the electricity consumption data of a building. If it is observed in a week, the difference in electricity consumption between weekdays and weekends can be clearly seen, which reflects a lower frequency periodic pattern. If the time granularity of observation is refined to every day, more detailed changes can be captured, such as a significant reduction in electricity consumption during lunch breaks and evening shifts. Therefore, different sampling frequencies not only affect the level of detail of the data, but also determine the periodic laws that can be observed. The convergence of periodic patterns on multiple time scales brings richer and more complex data features to the aggregated time series.
[0007] Spatial aggregation: In addition to temporal aggregation, spatial aggregation plays a key role in the aggregation of time series in multi-level scenarios. In these scenarios, data is gradually aggregated from the bottom layer of the scene to the top layer. This process involves the integration of information from different data sources, each of which may have a unique periodic pattern. As data flows from one level to another, the devices between the levels aggregate these data with different periodic characteristics and gradually form an aggregated time series. In this aggregation process, due to the fusion of data from different levels, the time-dependent components in the aggregated time series become complex and intertwined, forming an intricate time-dependent network.
[0008] like Figure 1 As shown in Figure 1, when these data are used for time series forecasting, the forecasting model needs to be able to learn and process these complex periodic patterns from the bottom layer and their superposition and fusion in the top layer data. This requires not only the model to have the ability to capture long-term dependencies, but also to be able to identify and separate the specific patterns and periodicity of data at each level, which is a major challenge in both model design and data processing.
[0009] Conventional long-term time series prediction methods are mostly designed and trained for time series collected by a single sensor. Aggregate time series in multi-level scenarios aggregate atomic time series collected by multiple single sensors in different local environments with different individual characteristics. These atomic time series have their own unique periodic patterns. After aggregation, these different periodic patterns are mixed and entangled in the aggregated time series, so that they show a certain chaotic tendency, which brings great challenges to prediction.
[0010] Existing time series forecasting technologies generally combine seasonal trend decomposition and deep learning models for long-term time series forecasting. However, they perform poorly when faced with aggregated time series formed by continuous aggregation in multi-level architectures and cannot make accurate forecasts. Specifically, current methods have the following disadvantages:
[0011] 1. Ignoring aggregated time series in multi-layer scenarios. Most of the current time series prediction methods are designed and trained for time series collected by a single sensor.
[0012] 2. Unable to handle the entanglement and overlap of periodic patterns in aggregated time series. The key to time series prediction is to discover and learn periodic patterns in sequence data, but the time series discovery mechanism of existing methods is affected by the entanglement and overlap of a large number of periodic patterns, and cannot accurately discover the periodic patterns contained in the aggregated time series.
[0013] 3. Currently, most time series forecasting methods use seasonal trend decomposition methods to reduce the impact of trend components in time series data, so that the model can focus on the discovery of periodic patterns. However, after seasonal trend decomposition of aggregated time series, there are still entanglements between the periodic patterns in the seasonal components, which brings challenges to prediction. Summary of the invention
[0014] The purpose of the present invention is to provide a method for multi-level scenario prediction time series, which can effectively deconstruct the intertwined high and low frequency components in the aggregated time series data, thereby eliminating the chaotic entanglement of periodic patterns between different frequencies in the data, and thus improve the accuracy and efficiency of time series prediction.
[0015] To achieve the above object, the present invention provides a method for multi-level scene prediction time series, comprising inputting the time series into a decoder and an encoder respectively;
[0016] Before the time series enters the decoder, it is first processed by the first data embedding module to obtain a high-dimensional vector of the embedding position and timing information, and the embedded high-dimensional vector is sent to the encoder to output the high-frequency component;
[0017] The encoder is provided with a first frequency decomposition module, a first frequency adaptive attention module, and a second frequency decomposition module in sequence;
[0018] Before the time series enters the encoder, it is first decomposed into a trend part and a seasonal part by the seasonal trend decomposition module. The second half of the trend part and the seasonal part will respectively constitute the first half of the data finally input into the decoder to provide the most recent historical information. The second half of the data finally input into the decoder is the data to be predicted, and placeholders are used for placeholding. The input data composed of the trend part is directly input into the decoder, and the input data composed of the seasonal part is first input into the second data embedding module to embed the position and timing information, and obtain the embedded high-dimensional vector, which is sent to the decoder to obtain the reconstructed time series.
[0019] The encoder is sequentially provided with a second frequency adaptive attention module, a third frequency decomposition module, an alignment frequency attention module, and a fourth frequency decomposition module.
[0020] Preferably, the historical time series input into the encoder will first be decomposed into two components, high frequency and low frequency, by the first frequency decomposition module, and the high frequency component will continue to be used for timing pattern discovery and learning using the first frequency adaptive attention module, and the intermediate potential timing component output by the first frequency adaptive module will be further decomposed by the second frequency decomposition module to obtain the high frequency component.
[0021] Preferably, the data input into the decoder will first be used for timing pattern discovery and learning by the second frequency adaptive attention module, and the intermediate potential timing components output by the second frequency adaptive module will be further decomposed by the third frequency decomposition module, and the third frequency decomposition module will extract low-frequency components and high-frequency components, wherein the low-frequency components extracted by the third frequency decomposition module are superimposed with the data composed of the trend part of the input according to the weight; at the same time, the high-frequency components that have been eliminated from the influence of the low-frequency components are sent to the alignment frequency attention module to discover and learn periodic patterns with the high-frequency components obtained from the encoder, and finally decomposed again by the fourth frequency decomposition module to obtain a reconstructed time series.
[0022] Preferably, the first data embedding module and the second data embedding module process data in the same way; the first frequency decomposition module, the second frequency decomposition module, the third frequency decomposition module and the fourth frequency decomposition module decompose in the same way, collectively referred to as FD; the first frequency adaptive attention module and the second frequency adaptive attention module periodically learn in the same way, collectively referred to as FAA.
[0023] Preferably, the seasonal trend decomposition module adopts the STL decomposition concept to decompose the time series into two components, trend and season. The input time series data is smoothed using the moving average method to extract the trend component. The seasonal component containing a complex periodic pattern is obtained by subtracting the trend component from the input original time series. The formula is as follows:
[0024]
[0025] here, and Represent the trend and seasonal components respectively, MA( ) represents the moving average operation to extract the trend part, Padding( ) represents the padding operation to ensure the consistency of the sequence length, Represents the input time series data.
[0026] Preferably, the calculation process of the encoder is formalized as follows:
[0027]
[0028] in, i∈{1,2} represents the low-frequency component and high-frequency component after the i-th FD module, respectively. FAA(·) represents the frequency adaptive module. Represents the timing data input into the encoder;
[0029] The encoders are stacked in multiple layers, and the output of the previous encoder layer is the input of the next encoder layer: Where l∈{1,…,N} represents the output of the first layer encoder.
[0030] Preferably, the calculation process of the decoder is formalized as follows:
[0031]
[0032]
[0033] in, Respectively represent the low-frequency component and high-frequency component after the i-th frequency decomposition module, i∈{1,2} represents the i-th low-frequency component The weight of , AFA( ) represents the alignment frequency attention module, represents the high frequency component of the input, represents the low-frequency component of the input, Represents the high-frequency component of the output, Represents the low-frequency component of the output;
[0034] The decoder also stacks multiple layers to adapt to more complex aggregated time series. There are N layers of decoders, and the aligned high-frequency latent variables from the corresponding 1-layer encoder are recorded as Then the output and input of the first layer of the decoder are Represents the low-frequency component of the output of the lth layer.
[0035] Preferably, the first frequency decomposition module, the second frequency decomposition module, the third frequency decomposition module and the fourth frequency decomposition module work in the following manner:
[0036] First, construct a variational problem. The corresponding constrained variational expression is:
[0037]
[0038] in, and represents the center frequency corresponding to the high-frequency component and the low-frequency component, δ(t is the Dirac function, j represents an imaginary number, t represents time, represents the decomposed high-frequency component, Represents the decomposed low-frequency component;
[0039] The Lagrange multiplier operator λ is introduced to transform the constrained variational problem into an unconstrained variational problem, and the ADMM iterative algorithm is used to obtain and The iteration formula is as follows:
[0040]
[0041] Here, α is the penalty factor that affects the bandwidth of each intrinsic modal component, ω represents the frequency, represents the low-frequency center frequency after the n+1th iteration, represents the high frequency center frequency after the n+1th iteration, Corresponding to λ(ω), The Fourier transform of and The iterative formula performs iterations and continues to update until ε is a minimum value.
[0042] Preferably, the first frequency adaptive attention module and the second frequency adaptive attention module work in the following manner:
[0043] Set the time series in the time domain Contains data points (x1, x2, ..., x t ), through fast Fourier transform, this time series is converted into f=T / / 2,each is a complex number representing the amplitude and phase of each Fourier component. The frequency-adaptive attention module calculates The average amplitude in D dimensions and select the M Fourier components with larger amplitudes. This process is formalized as:
[0044]
[0045] Here, FFT(·) represents the fast Fourier transform, Avg(·) is responsible for calculating the average value of the amplitude in D dimensions, the amplitude is calculated by Amp(·), M∈C M×D M Fourier components representing the maximum amplitude;
[0046] The obtained M is multiplied by a linear layer P, and the weight of P is updated as the model is trained. The result of MP is converted back to the time domain through the inverse Fourier transform as the output of the frequency adaptive attention module, which is formalized as follows:
[0047] FAA(x)=FFT -1 (MP).
[0048] Preferably, the alignment frequency attention module works as follows:
[0049] The M main Fourier components with the largest amplitude are selected as the query, key, and value in the attention mechanism, which are represented as:
[0050] Q = Select-M (FFT (q))
[0051] K = Select-M (FFT (k))
[0052] V=Select-M(FFT(v))
[0053] Among them, Q, K, and V∈C M×D , represents the main Fourier components after selection. After determining these main components, the alignment frequency attention module will proceed to the next step, which is defined as:
[0054] AFA(q,k,v)FFT -1 =Padding(σ((Q·K T )·V))
[0055] Here, σ represents the activation function, which transforms these weighted Fourier components in the frequency domain back to the time domain, thereby reconstructing the original time series.
[0056] Therefore, the present invention adopts the above-mentioned method for multi-level scene prediction time series, and the technical effects are as follows:
[0057] (1) This paper focuses on the often neglected task of aggregate time series prediction, analyzes the differences in periodic patterns between atomic time series and aggregate time series, deeply studies the chaotic characteristics in aggregate time series data, and designs a set of methods for measurement to guide the design of subsequent models.
[0058] (2) Different from the traditional seasonal trend decomposition method, the present invention proposes a more sophisticated and applicable frequency decomposition module built into the proposed AnaNET to address the problem of pattern entanglement that frequently occurs in aggregated time series data, so as to deal with the problem of pattern entanglement that still exists even after seasonal trend decomposition.
[0059] (3) In order to more effectively handle the convergence process of aggregated time series, this paper designs a progressive reverse dissection framework. This framework continuously performs frequency decomposition and explores temporal dependencies along the reverse path of data convergence in multi-layer scenarios. This allows AnaNET to focus more on processing temporal dependencies in different frequency components, thereby improving the accuracy and interpretability of predictions.
[0060] (4) When discovering time series dependencies, the AnaNET model uses fast Fourier decomposition to convert data from the time domain to frequency and learns the Fourier components to reconstruct the time series data. Different from other methods that also perform frequency learning, the present invention only selects some Fourier components to reduce computational overhead, specifically, selects the M components with the largest amplitude, ensuring that key information is not lost while ensuring computational efficiency.
[0061] (5) The AnaNET model proposed in the present invention is different from other time series prediction models. The internal structure of the model is simple and efficient, with frequency as the main line, and only includes two parts: frequency decomposition module and frequency dependency discovery module. In addition, before entering the model, the data is decomposed into seasonal trends so that AnaNET can focus more on the complex seasonal part. The present invention uses seasonal trend decomposition and frequency decomposition in combination, which can give play to the advantages of different decomposition methods, thereby achieving more accurate predictions. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 It is the aggregation process of time series in multi-level power scenarios;
[0063] Figure 2 Embedding graph for time series data;
[0064] Figure 3 Calculation process for frequency adaptive attention module;
[0065] Figure 4 Calculation process for the alignment frequency attention module;
[0066] Figure 5 This is the AnaNET model architecture diagram;
[0067] Figure 6 Predictions for different lengths are shown; Figure 6 (a) Predict the next 336 points for MD_manu; Figure 6 (b) Predict the next 96 points for MD_food; Figure 6 (c) Predict the next 96 points for MD_phar; Figure 6 (d) Forecast the next 196 points for ETTm2. DETAILED DESCRIPTION
[0068] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.
[0069] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.
[0070] Embodiment 1
[0071] The explanations of some special terms mentioned in this embodiment are as follows:
[0072] PyTorch: PyTorch is an open source deep learning framework developed by Facebook's artificial intelligence research team. After its release in 2016, PyTorch quickly became popular in the research community for its ease of use, flexibility, and powerful functions.
[0073] Seasonal and Trend decomposition (STL): STL decomposition (Seasonal and Trend decomposition using Loess) is a decomposition algorithm widely used in time series data. Its main purpose is to deconstruct time series into three basic components: trend, seasonality, and residual. The algorithm gradually fits and identifies different components in the time series through the method of local weighted regression (Loess), and is particularly suitable for data sets containing nonlinear trends and significant seasonal changes.
[0074] Self-Attention: Self-attention is a special type of attention mechanism. The attention mechanism is a method that mimics the human visual and cognitive system, which allows the neural network to focus on the relevant parts when processing input data. By introducing the attention mechanism, the neural network can automatically learn and selectively focus on the important information in the input, improving the performance and generalization ability of the model. The basic idea of the self-attention mechanism when processing sequence data is that each element can be associated with other elements in the sequence, rather than just relying on elements in adjacent positions.
[0075] The present invention provides a method for predicting time series in multi-level scenarios, which is mainly aimed at the long-term prediction task of aggregated time series in multi-layer scenarios. It is an end-to-end Encoder-Decoder model, adopts a multi-layer framework, and the number of layers can be adjusted according to the complexity of the scene. The algorithm mainly includes a data embedding module, a seasonal trend decomposition module, a frequency decomposition module, a frequency adaptive attention module and an aligned frequency attention module.
[0076] The specific steps include:
[0077] Step 1: Dataset creation
[0078] In the data set establishment step, the following three parts need to be completed: data collection, data preprocessing, data analysis and quantification.
[0079] Step 1.1. Data Collection
[0080] The present invention collects multi-level power energy consumption data from three real industrial parks. These parks have enterprises from all walks of life (including but not limited to pharmaceutical, food, manufacturing), multiple buildings and various heterogeneous equipment. As shown in Table 1, the power energy consumption data collected by the present invention can be generally classified according to three levels: the five categories under the first-level classification are lighting electricity, production electricity, special electricity, power electricity and heating electricity, the second-level classification has 15 categories, and the third-level classification has 4 categories. There are a total of 287 points under the three levels. Finally, these points are aggregated into three complex aggregated time series data sets MD_manu, MD_pahr, and MD_food. Combined with the well-known open source data sets ECL and ETTm2 in the power field and ETT_MIX formed by the aggregation of ETTm, a total of 6 data sets are used as the data sets of the present invention.
[0081] Table 1 Power collection points
[0082]
[0083] Step 1.2: Data preprocessing
[0084] The collected data are preprocessed, including but not limited to using exponential smoothing to process abnormal values and outliers, using scaling to process inconsistent magnitudes before and after, and using linear difference and nearest neighbor interpolation to fill missing values.
[0085] Step 1.3: Data analysis
[0086] The collected data is analyzed to understand its general distribution and regularity. The means of analysis include but are not limited to: in order to determine whether the time series sequence follows a specific distribution within a specified period, the time series data is reconstructed according to daily, monthly and other periods for visual analysis, so as to facilitate the preliminary exploration of the periodicity of the data. The maximum value plays a key role in many tasks. The time series data is reconstructed according to the maximum or minimum value of the daily, weekly and other periods to draw the maximum value curve to observe the fluctuation of the upper and lower limits of electricity within the period. Time series sequences usually show different patterns on weekdays and holidays. The time series data is reconstructed according to weekdays and holidays to observe the difference in fluctuation patterns between the two. The present invention further reconstructs and draws according to working hours and non-working hours to obtain more fine-grained insights.
[0087] Step 1.4: Quantify data features
[0088] In the multi-level scenario, there are a large number of time series, which are constantly converging upward to form aggregated time series. The number of periodic patterns contained in the aggregated time series and the degree of chaos caused by the entanglement of these patterns have a great impact on the accuracy of the prediction. Therefore, the present invention uses, including but not limited to, the Lyapunov index to measure the degree of chaos of the time series data, and uses fast Fourier transform to measure the main periodic frequency contained in the time series data. The specific steps are as follows:
[0089] Step 1.4.1: Using Lyapunov exponents to measure chaotic characteristics
[0090] The time series data is reconstructed into phase space by delay method to adapt to the calculation of Lyapunov coefficients to reveal the degree of chaos of the data. This step involves two important parameters, namely the sequence delay time τ and the embedding dimension m of the phase space.
[0091] The sequence delay time τ is calculated by the cross-correlation function. For a single time series, the optimal τ is found by calculating the cross-correlation between the sequence and itself at different delays. Its autocorrelation function R(τ) at a delay of τ is defined as:
[0092]
[0093] in, is the mean of the time series, and N is the length of the time series.
[0094] Using the above formula, τ can be determined. The calculated autocorrelation function R(τ) can show the repetitive nature of the time series itself as it changes with τ. The ideal τ value usually corresponds to the first local minimum or zero crossing point of R(τ). The τ selected by the present invention is the first point in the time series where the autocorrelation begins to decrease significantly.
[0095] The embedding dimension m is determined using the pseudo nearest neighbor method, aiming to find a dimension under which the dynamics of the time series can be fully unfolded.
[0096] Step 1.4.2: Use Fourier transform to measure the number of main frequency components
[0097] In time series forecasting, it is crucial to deeply understand and mine the periodic frequencies in time series data. These periodic frequencies often reveal the laws and dynamics behind the data. Measuring the main frequencies can provide key information for further data analysis and model building.
[0098] Fourier transform is a mathematical tool widely used in time series data analysis. It converts time series data from the time domain to the frequency domain, allowing the present invention to identify and quantify the main frequency quantity in the data.
[0099] In this process, the original time series data First, it is divided into T segments, each segment is divided into two segments. t Indicates that each segment contains J data points. Next, perform FFT on each segment of data, according to the following formula
[0100]
[0101] The frequency component F(ω) is calculated. The present invention calculates the K frequency components with the largest amplitude in each segment, which are regarded as the most important frequencies in the time series slice of this segment. The component with the largest amplitude is selected to capture the frequency components that have the greatest impact on the analysis. Afterwards, the present invention counts the frequency of each selected frequency component in all T segments. It aims to find the main frequency components that appear consistently in the entire time series data set. If the number of main frequencies is large, it means that the time series sequence requires more decomposition and time series dependency learning. Finally, this paper adopts a threshold screening method to select frequency components with a frequency higher than θG, where θ is a preset frequency threshold ratio. This screening is based on the following formula
[0102]
[0103] Here, K is the number of maximum amplitude components selected from all segments. This process ensures that only those frequency components that appear frequently in the entire data set and are therefore most likely to represent the inherent periodic characteristics of the data are retained.
[0104] The final measurement results of the characteristics of the collected data sets are shown in Table 2.
[0105] Table 2 Dataset feature measurement results
[0106]
[0107]
[0108] Step 2: AnaNET model
[0109] The model established by the present invention will input the time series data of the input model into the encoder part and the encoder part respectively. Among them, the time series data input into the encoder will be converted into a 256-dimensional high-dimensional vector by the data embedding module, so that the encoder can learn the time series pattern in the high-dimensional space. The data to be input into the decoder will first pass through the seasonal trend decomposition module to decompose it into trend and seasonal components. The second half of the trend and seasonal components will constitute the first half of the final input data to provide the most recent historical time series information. The second half of the final input data is the data to be predicted, and placeholders are used to hold the placeholders to achieve direct multi-step prediction. Specifically including
[0110] Input the time series into the decoder and encoder respectively;
[0111] Before the time series enters the decoder, it is first processed by the first data embedding module to obtain a high-dimensional vector of the embedding position and timing information, and the embedded high-dimensional vector is sent to the encoder to output the high-frequency component;
[0112] Encoder: The encoder is provided with a first frequency decomposition module, a first frequency adaptive attention module and a second frequency decomposition module in sequence. The historical time series input into the encoder will first be decomposed into two components, high frequency and low frequency, by the first frequency decomposition module. Among them, for the high frequency components with more periodic and fluctuation information, the first frequency adaptive attention module is used to discover and learn the timing pattern, and the intermediate potential timing components output by the first frequency adaptive module are further decomposed to eliminate the interference of the low frequency components, so as to guide the encoder to predict the high frequency information more accurately. The calculation process of the above encoder can be formalized as:
[0113]
[0114] in, i∈{1,2} respectively represent the low-frequency and high-frequency components after the i-th FD module. FAA(·) represents the frequency adaptive module. The encoder can be stacked in multiple layers, and the output of the previous encoder layer is the input of the next encoder layer: Where l∈{1,…,N} represents the output of the first layer encoder.
[0115] Before the time series enters the encoder, it is first decomposed into a trend part and a seasonal part by the seasonal trend decomposition module. The second half of the trend part and the seasonal part will respectively constitute the first half of the data finally input into the decoder to provide the most recent historical information. The second half of the data finally input into the decoder is the data to be predicted, and placeholders are used for placeholding. The input data composed of the trend part is directly input into the decoder, and the input data composed of the seasonal part is first input into the second data embedding module to embed the position and timing information, and obtain the embedded high-dimensional vector, which is sent to the decoder to obtain the reconstructed time series.
[0116] The encoder is sequentially provided with a second frequency adaptive attention module, a third frequency decomposition module, an alignment frequency attention module, and a fourth frequency decomposition module.
[0117] Before the historical time series is input into the decoder, the data is subjected to seasonal trend decomposition (STL) to extract its trend and seasonal components. The second half of the trend and seasonal components is used as the first half of the final input to provide the most recent historical time series information, and the second half to be predicted is occupied by placeholders to achieve direct multi-step prediction. The decoder consists of frequency decomposition (FD), frequency adaptive attention module (FAA) and aligned frequency attention module (AFA). The FD module continuously extracts low-frequency components and high-frequency components, where the low-frequency components extracted by each layer of FD are superimposed with the input trend part according to certain weights. At the same time, the high-frequency components that are eliminated from the influence of the low-frequency components are sent to the aligned frequency attention module for temporal dependency discovery with the high-frequency components aligned in the encoder, and the temporal pattern is captured using AFA. The calculation process of the decoder can be formalized as follows:
[0118]
[0119] in Represent the low-frequency and high-frequency components after the i-th frequency decomposition module respectively. Represents the i-th low-frequency component The weight of , AFA() represents the alignment frequency attention module, represents the high frequency component of the input, represents the low-frequency component of the input, Represents the high-frequency component of the output, Represents the low-frequency component of the output.
[0120] The decoder can also be stacked in multiple layers to accommodate more complex aggregated temporal sequences. Assuming there are N layers of decoders, the aligned high-frequency latent variables from the corresponding 1-layer encoder are denoted as Then the output and input of the first layer of the decoder are Represents the low-frequency component of the output of the lth layer.
[0121] Data embedding module: In the present invention, numerical embedding, position embedding, and time series information embedding are used in combination for data embedding, so that the model can discover and learn the most critical elements and dependencies in relatively rich information. In terms of time series information embedding, in addition to the commonly used information such as week, month, year, and holiday, the present invention further adds finer-grained features such as minutes and hours to facilitate more detailed modeling. It is worth noting that the dimension of minute embedding is set according to the sampling frequency of the data, that is, sampling every 15 minutes is set to 4, sampling every 5 minutes is set to 12, and so on. The final data embedding is as follows: Figure 2 shown.
[0122] The seasonal trend decomposition module adopts the STL (Seasonal and Trend decomposition using Loess) decomposition concept. The difference from the original STL decomposition is that the present invention only decomposes the time series into two components, trend and season, and does not further decompose the residual component. The present invention uses the moving average method to smooth the input time series data to extract the trend component. The original input sequence minus the trend component is the seasonal component containing a complex periodic pattern. The formula is as follows:
[0123]
[0124] here, and Represent the trend and seasonal components respectively, MA() represents the moving average operation to extract the trend part, and Padding() represents the padding operation to ensure the consistency of the sequence length. Represents the input time series data, Represents the time series data input into the encoder. By applying a moving average (MA(·)) and combining a padding operation (Padding), the module maintains the consistency of the sequence length.
[0125] Multi-level scenarios show a tendency to converge periodic patterns in both time and space dimensions. In these convergence processes, periodic patterns of various frequencies are intertwined and overlapped, resulting in more entangled periodic patterns in the aggregated time series, thus showing chaotic characteristics. In order to enhance the ability of subsequent time dependency discovery, the present invention designs a frequency-centric decomposition method and embeds it into the end-to-end model AnaNET. This module can decompose the high-frequency and low-frequency components within the seasonal part and eliminate the mutual interference between them. Therefore, this enables AnaNET to focus more on learning the time dependency in the high-frequency component, thereby improving the accuracy of its prediction.
[0126] Specifically, based on the computational principle of variational mode decomposition, the present invention uses Pytorch to develop a frequency decomposition module (FD). This module is specially customized for multidimensional time series decomposition and reduces its computational overhead, and can be losslessly embedded as a module in a deep model. In addition, a significant advantage of the frequency decomposition module designed by the present invention is that it can perform decomposition operations without knowing the core frequencies in the time series data in advance, thereby significantly enhancing its adaptability to unknown data sets.
[0127] The frequency decomposition module designed by the present invention first needs to construct a variational problem. Assume that the input time series will be decomposed into high frequency components and low frequency components You must ensure and is a limited bandwidth intrinsic modal component with an independent center frequency. and The sum of the estimated bandwidths of is minimized, subject to the constraint that and The sum is equal to the original time series The corresponding constrained variational expression is:
[0128]
[0129] in, and represents the center frequency corresponding to the high-frequency component and the low-frequency component. δ(t) is the Dirac function, j represents an imaginary number, t represents time, represents the decomposed high-frequency component, represents the low-frequency component of the decomposition. The Lagrange multiplier operator λ is introduced to transform the constrained variational problem into an unconstrained variational problem, and the ADMM iterative algorithm is used to obtain and The iterative formula is as follows:
[0130]
[0131] Here, α is the penalty factor that affects the bandwidth of each intrinsic modal component, ω represents the frequency, represents the low-frequency center frequency after the n+1th iteration, Represents the high frequency center frequency after the n+1th iteration. and Corresponding to The Fourier transform of λ(ω). We will iterate five times through equations (4-8), updating them continuously until ε is a minimum value and is set to 5e-5.
[0132] The function of the frequency adaptive attention module (FAA) is to find the main frequencies in the high-frequency components to learn the temporal patterns in the data. To achieve this, the present invention uses the fast Fourier transform (FFT) to convert the high-frequency components containing complex periodic patterns decomposed by the frequency decomposition module (FD) into the frequency domain to better discover the periodic frequencies contained therein and improve the accuracy of model prediction.
[0133] A key issue when using FFT is which Fourier components should be selected to represent the time series. Selecting all Fourier components often leads to overfitting and reduced computational efficiency. A common practice is to make random selections. Although random selections ensure computational efficiency, there is still a possibility of missing important frequencies, resulting in the loss of important information. In the FAA designed by the present invention, it is preferred to adaptively select M main frequency components from the Fourier components of different data to ensure that key information is not lost while maintaining computational efficiency.
[0134] Specifically, if Figure 3 As shown, suppose there is a time series in the time domain Contains data points (x1, x2, ..., x t ). Through the fast Fourier transform, this time series can be converted to f = T / / 2. Due to the conjugation of the frequency domain, the present invention only considers the first half of the Fourier component. is a complex number representing the amplitude and phase of each Fourier component. The frequency-adaptive attention module calculates The average amplitude in D dimensions, and select M Fourier components with larger amplitudes. The reason why the present invention uses amplitude as a screening criterion is that high-amplitude Fourier components often determine the final value of time series data. Therefore, by selecting these components, FAA can capture the most important characteristics of the time series. This process can be formalized as:
[0135] M=Select-M(Avg(Amp(FFT(x))))
[0136] Here, FFT(·) represents the fast Fourier transform, and Avg(·) is responsible for calculating the average value of the amplitude in D dimensions, which is calculated by Amp(·). M×D The M Fourier components representing the maximum amplitude.
[0137] Here, in order to let FAA learn the weights between different Fourier components, this paper multiplies the obtained M with a linear layer P, and the weight of P can be updated as the model is trained. The result of MP will be converted back to the time domain through the inverse Fourier transform as the output of the frequency adaptive attention module, which is formalized as follows:
[0138] FAA(x)=FFT -1 (MP).
[0139] Aligned Frequency Attention Module: When performing time series prediction tasks, especially those involving high-frequency time series, the Attention mechanism in the classic Transformer architecture is difficult to better capture and utilize high-frequency periodic information. To this end, the present invention proposes an Aligned Frequency Attention (AFA) module. Different from the classic Transformer, which only inputs the information of the last layer of encoder into the decoder, AFA aligns the high-frequency component latent variables of the response from the encoders at the corresponding level, and uses the information of this latent variable to guide the recognition and learning of high-frequency periodic patterns.
[0140] like Figure 4 As shown, in the aligned frequency attention module, the query (query, q) of the self-attention mechanism comes from the decoder, representing the embedded time domain time series In addition to the query, the key (key, k) and value (value, v) are derived from the corresponding high-frequency latent variables in the encoder, denoted by k∈R T×D and v∈R T×D Similar to the process of the frequency adaptive attention module, the aligned frequency attention module also uses fast Fourier transform (FFT) to convert these queries, keys, and values from the time domain to the frequency domain, and then discovers and learns periodic patterns in the frequency domain.
[0141] The most representative periodic pattern in the frequency domain usually corresponds to the Fourier component with the largest amplitude, so this paper selects M main Fourier components with the largest amplitude as the query, key and value in the attention mechanism. These components are represented as:
[0142] Q = Select-M (FFT (q))
[0143] K = Select-M (FFT (k))
[0144] V=Select-M(FFT(v))
[0145] Among them, Q, K, and V∈C M×D , represents the main Fourier components after selection. Once these main components are determined, the alignment frequency attention module will proceed to the next step, which is defined as:
[0146] AFA(q,k,v)FFT -1 =Padding(σ((Q·KT)·V))
[0147] Here, σ represents the activation function, and Padding ensures that the output after the inverse fast Fourier transform (IFFT) operation matches the dimension of the original time series. Through IFFT, the present invention converts these weighted Fourier components in the frequency domain back to the time domain, thereby reconstructing the original time series.
[0148] Step 3: Model training
[0149] The prediction length of the present invention is more than 96 points in the future, which is in line with the long sequence prediction category recognized by the industry. The segmentation of the data set used follows the industry practice and is divided into training set, validation set and test set at a ratio of 7:1:2. In order to adapt to data sets of various sizes and distributions, the present invention uniformly adopts a standardized method for data preprocessing to accelerate the convergence speed during model training. In terms of parameter setting, according to the results of preliminary experiments, the alpha value of the frequency decomposition module is set to 100, while the parameter M in the frequency adaptive attention module and the aligned frequency attention module is fixed to 64.
[0150] The present invention selects the mean square error commonly used in time series prediction and mean absolute error As the loss function, the performance of the designed model is comprehensively evaluated from the perspectives of squared deviation and absolute deviation.
[0151] In terms of model structure, the embedding dimension is set to 256, and a structure consisting of 2 encoder layers and 1 decoder layer is adopted. In terms of optimization strategy, the training process uses the Adam optimizer and sets the initial learning rate to 10 -4 During the training process, the learning rate is halved after each cycle is completed to finely control the learning progress. The batch size is set to 32, and the model is trained for a total of 20 rounds. An early stopping mechanism is introduced. If there is no performance improvement after 3 consecutive rounds of training, the training is terminated early.
[0152] All experiments were implemented using the PyTorch framework (version 1.9.0, Python version 3.6.8) on a computer equipped with dual NVIDIA GeForce RTX 3090 GPUs.
[0153] The prediction effect of the final model on the collected data set is shown in Table 3 and Figure 6 As shown in the figure, on 6 data sets, compared with the most advanced long time series prediction algorithm in the industry, the MSE and MAE indicators have been improved by an average of about 20%.
[0154] Table 3 Model performance display
[0155] Dataset type Chaos Index Main frequency MES Performance MAE Performance MD_manu AggTS 0.013962 68 0.178 0.320 MD_phar AggTS 0.013462 75 0.141 0.260 MD_food AggTS 0.012953 45 0.325 0.423 MD_MIX AggTS 0.007460 28 0.042 0.156 ECL AtoTS 0.006238 30 0.064 0.192 ETTm2 AtoTS 0.006130 24 0.274 0.387
[0156] Therefore, the present invention adopts the above-mentioned method for predicting time series in multi-level scenarios, focusing on the prediction task of aggregated time series. The present invention deeply analyzes the generation and aggregation process of time series in multi-layer scenarios, classifies them into atomic time series and aggregated time series, and summarizes the process of time series aggregation in the dual dimensions of time and space, providing a theoretical basis for the prediction of aggregated time series. The innovative frequency decomposition module is different from the traditional seasonal trend decomposition method. The present invention designs a more refined and applicable frequency decomposition module for aggregated time series, and embeds it in the deep learning model, which effectively solves the problem of entanglement and overlap of periodic patterns in aggregated time series. Progressive reverse dissection framework: The present invention designs a progressive reverse dissection framework according to the process of time series convergence in multi-layer scenarios, follows the reverse path of the convergence process, gradually performs frequency decomposition and captures the time series dependencies in the data, thereby achieving more accurate predictions. Based on the above advantages, the method proposed by the present invention has obvious advantages in processing aggregated time series data, and can effectively deconstruct the interwoven high and low frequency components in the aggregated time series data, thereby eliminating the chaotic entanglement of periodic patterns between different frequencies in the data, thereby improving the accuracy and efficiency of time series prediction.
[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.
Claims
1. A method for multi-level scenario prediction time series, characterized in that: include: Input the time series into the decoder and encoder respectively; Before the time series enters the decoder, it is first processed by the first data embedding module to obtain a high-dimensional vector of the embedding position and timing information, and the embedded high-dimensional vector is sent to the encoder to output the high-frequency component; The encoder is provided with a first frequency decomposition module, a first frequency adaptive attention module, and a second frequency decomposition module in sequence; Before the time series enters the encoder, it is first decomposed into a trend part and a seasonal part by the seasonal trend decomposition module. The second half of the trend part and the seasonal part will respectively constitute the first half of the data finally input into the decoder to provide the most recent historical information. The second half of the data finally input into the decoder is the data to be predicted, and placeholders are used for placeholding. The input data composed of the trend part is directly input into the decoder, and the input data composed of the seasonal part is first input into the second data embedding module to embed the position and timing information, and obtain the embedded high-dimensional vector, which is sent to the decoder to obtain the reconstructed time series. The encoder is sequentially provided with a second frequency adaptive attention module, a third frequency decomposition module, an alignment frequency attention module, and a fourth frequency decomposition module.
2. A method for multi-level scenario prediction time series according to claim 1, characterized in that: The historical time series input into the encoder will first be decomposed into high-frequency and low-frequency components by the first frequency decomposition module, and the high-frequency component will continue to be used for timing pattern discovery and learning using the first frequency adaptive attention module, and the intermediate potential timing component output by the first frequency adaptive module will be further decomposed by the second frequency decomposition module to obtain the high-frequency component.
3. A method for multi-level scenario prediction time series according to claim 1, characterized in that: The data input into the decoder will first be discovered and learned by the second frequency adaptive attention module for timing pattern discovery, and the intermediate potential timing components output by the second frequency adaptive module will be further decomposed by the third frequency decomposition module. The third frequency decomposition module extracts low-frequency components and high-frequency components, where the low-frequency components extracted by the third frequency decomposition module are superimposed with the data composed of the trend part of the input according to the weight; at the same time, the high-frequency components that have been eliminated from the influence of the low-frequency components are sent to the alignment frequency attention module to discover and learn periodic patterns with the high-frequency components obtained from the encoder, and finally decomposed again by the fourth frequency decomposition module to obtain the reconstructed time series.
4. A method for multi-level scenario prediction time series according to claim 1, characterized in that: The first data embedding module and the second data embedding module process data in the same manner; the first frequency decomposition module, the second frequency decomposition module, the third frequency decomposition module and the fourth frequency decomposition module decompose in the same manner and are collectively referred to as FD; The first frequency adaptive attention module and the second frequency adaptive attention module have the same periodic learning method and are collectively referred to as FAA.
5. The method for multi-level scenario prediction time series according to claim 1, characterized in that: The seasonal trend decomposition module adopts the STL decomposition concept to decompose the time series into two components: trend and season. The moving average method is used to smooth the input time series data to extract the trend component. The seasonal component containing complex periodic patterns is obtained by subtracting the trend component from the input original time series. The formula is as follows: here, and Represent the trend and seasonal components respectively, MA( ) represents the moving average operation to extract the trend part, Padding( ) represents the padding operation to ensure the consistency of the sequence length, Represents the input time series data.
6. A method for multi-level scenario prediction time series according to claim 1, characterized in that: The calculation process of the encoder is formalized as: in, They represent the low-frequency component and high-frequency component after the i-th FD module, respectively. FAA(·) represents the frequency adaptive module. Represents the timing data input into the encoder; The encoders are stacked in multiple layers, and the output of the previous encoder layer is the input of the next encoder layer: Where l∈{1,…,N} represents the output of the first layer encoder.
7. A method for multi-level scenario prediction time series according to claim 1, characterized in that: The computation process of the decoder is formalized as follows: in, Respectively represent the low-frequency component and high-frequency component after the i-th frequency decomposition module, Represents the i-th low-frequency component The weight of , AFA( ) represents the alignment frequency attention module, represents the high frequency component of the input, represents the low-frequency component of the input, Represents the high-frequency component of the output, Represents the low-frequency component of the output; The decoder also stacks multiple layers to adapt to more complex aggregated time series. There are N layers of decoders, and the aligned high-frequency latent variables from the corresponding 1-layer encoder are recorded as Then the output and input of the first layer of the decoder are Represents the low-frequency component of the output of the lth layer.
8. A method for multi-level scenario prediction time series according to claim 1, characterized in that: The working mode of the first frequency decomposition module, the second frequency decomposition module, the third frequency decomposition module and the fourth frequency decomposition module is: First, construct a variational problem. The corresponding constrained variational expression is: in, and represents the center frequency corresponding to the high-frequency component and the low-frequency component, δ(t is the Dirac function, j represents an imaginary number, t represents time, represents the decomposed high-frequency component, Represents the decomposed low-frequency component; The Lagrange multiplier operator λ is introduced to transform the constrained variational problem into an unconstrained variational problem, and the ADMM iterative algorithm is used to obtain and The iteration formula is as follows: Here, α is the penalty factor that affects the bandwidth of each intrinsic modal component, ω represents the frequency, represents the low-frequency center frequency after the n+1th iteration, represents the high frequency center frequency after the n+1th iteration, Corresponding to λ(ω), The Fourier transform of and The iterative formula performs iterations and continues to update until ε is a minimum value.
9. A method for multi-level scenario prediction time series according to claim 1, characterized in that: The working modes of the first frequency adaptive attention module and the second frequency adaptive attention module are as follows: Set the time series in the time domain Contains data points (x1, x2, ..., x t ), through fast Fourier transform, this time series is converted into Each is a complex number representing the amplitude and phase of each Fourier component. The frequency-adaptive attention module calculates The average amplitude in D dimensions and select the M Fourier components with larger amplitudes. This process is formalized as: Here, FFT(·) represents the fast Fourier transform, Avg(·) is responsible for calculating the average value of the amplitude in D dimensions, the amplitude is calculated by Amp(·), M∈C M×D M Fourier components representing the maximum amplitude; The obtained M is multiplied by a linear layer P, and the weight of P is updated as the model is trained. The result of MP is converted back to the time domain through the inverse Fourier transform as the output of the frequency adaptive attention module, which is formalized as follows: FAA(x)=FFT -1 (MP)。 10. The method for multi-level scenario prediction time series according to claim 1, characterized in that: The alignment frequency attention module works as follows: The M main Fourier components with the largest amplitude are selected as the query, key, and value in the attention mechanism, which are represented as: Q = Select-M (FFT (q)) K = Select-M (FFT (k)) V=Select-M(FFT(v)) Among them, Q, K, and V∈C M×D , represents the main Fourier components after selection. After determining these main components, the alignment frequency attention module will proceed to the next step, which is defined as: AFA(q,k,v)FFT -1 =Padding(σ((Q·K T )·V)) Where σ represents the activation function, which transforms these weighted Fourier components in the frequency domain back to the time domain, thereby reconstructing the original time series.
Citation Information
Cited By
Time sequence prediction method and device based on MPDLINE model and medium
CN120144960A
Face image data encryption method and system based on deep learning
CN120822225A
