A long time series prediction method based on time-frequency variation pattern modeling

Abstract

The application provides a long-time sequence prediction method based on time-frequency change mode modeling, and the method comprises the following steps: performing feature coding on an original time sequence to obtain a coded time sequence; inputting the coded time sequence into a time sequence prediction model for training to obtain a predicted time sequence output by the time sequence prediction model; obtaining a total loss of the time sequence prediction model based on a time domain loss and a frequency domain loss; updating a structure parameter of the time sequence prediction model based on the total loss to obtain an updated time sequence prediction model, and continuously training the coded time sequence based on the updated time sequence prediction model until a preset training number is reached to obtain a trained time sequence prediction model; obtaining a coded time sequence corresponding to an application scenario; and obtaining a predicted time sequence corresponding to the application scenario.

Description

Technical Field

[0001] This invention relates to the field of time series data modeling technology, and in particular to a long-term series prediction method based on time-frequency variation pattern modeling. Background Technology

[0002] Time series data refers to data recorded by observing certain things or phenomena at a certain sampling frequency within a certain time range. Time series data prediction, on the other hand, models long-term dependency features to predict future development trends. Currently, using deep learning technology for time series data prediction is the mainstream research approach, leading to the emergence of many advanced methods. For example, methods based on recurrent neural networks (RNNs) have very flexible nonlinear modeling capabilities, but they are prone to gradient vanishing and exploding problems at long time steps. Convolutional neural network (CNN) methods are limited by 1D convolution feature extraction, and can only model data between adjacent time points, failing to address long-term dependencies. Transformer-based methods use attention mechanisms for feature extraction and temporal relationship modeling; however, time series data contains abstract, deep dependencies, making it very difficult to model stable and reliable dependencies from discrete time points.

[0003] Therefore, the rich and complex temporal variation patterns contained in time series data severely limit the fine-grained semantic analysis of them by existing methods. Summary of the Invention

[0004] This invention provides a long-term series prediction method based on time-frequency variation pattern modeling, which can solve the technical problem that the prediction effect of time series data is poor due to the coupling of rich and complex time variation patterns in time series data.

[0005] This invention provides a long-term series prediction method based on time-frequency variation pattern modeling, the method comprising:

[0006] The original time series is feature-encoded to obtain the encoded time series.

[0007] The encoded time series is input into a time series prediction model based on time-frequency variation patterns for training, resulting in the predicted time series output by the model. The time series prediction model comprises L residually connected spectral feature extraction and fusion modules. During each training iteration, the input and output data of the current spectral feature extraction and fusion module are fused, and the fusion result is used as the input data for the next module. The input data for the first module is the encoded time series, and the fusion result of the output data and input data of the last module is the predicted time series output by the model.

[0008] The time-domain loss of the time series prediction model is obtained based on the original time series and the predicted time series;

[0009] Fourier transforms are performed on the original time series and the predicted time series respectively. The frequency domain loss of the time series prediction model is obtained based on the original time series and the predicted time series after the Fourier transform.

[0010] The total loss of the time series prediction model is obtained based on the time domain loss and the frequency domain loss.

[0011] The structural parameters of the time series prediction model are updated based on the total loss to obtain the updated time series prediction model. The encoded time series is then trained based on the updated time series prediction model until the preset number of training iterations is reached to obtain the trained time series prediction model.

[0012] Collect time series corresponding to the application scenario, encode the features of the time series corresponding to the application scenario, and obtain the encoded time series corresponding to the application scenario.

[0013] Based on the trained time series prediction model, the encoded time series corresponding to the application scenario is predicted to obtain the predicted time series corresponding to the application scenario.

[0014] Preferably, the fusion result of the input and output data of the current spectral feature extraction and fusion module is obtained by the following formula:

[0015]

[0016] In the formula, This represents the input data for the l-th spectral feature extraction and fusion module. This represents the output data of the l-th spectral feature extraction and fusion module. This represents the fusion result of the input and output data of the l-th spectral feature extraction and fusion module.

[0017] Preferably, each spectral feature extraction and fusion module includes a wavelet transform sub-module, a multi-scale parallel 2D convolutional network and a sparse self-attention feature fusion module network connected in series, and each multi-scale parallel 2D convolutional network includes two parallel convolutional sub-networks connected in series.

[0018] The wavelet transform submodule is used to perform wavelet transform on the input data of the current spectral feature extraction and fusion module to obtain the wavelet-transformed data;

[0019] A multi-scale parallel 2D convolutional network is used to extract features from the wavelet transform data to obtain the feature-extracted data.

[0020] A sparse self-attention feature fusion module network is used to fuse the features extracted from the data to obtain the fused data, which is the output data of the current spectral feature extraction and fusion module.

[0021] Preferably, the wavelet transform submodule is used to perform wavelet transform on the input data of the current spectral feature extraction and fusion module, and the wavelet transformed data includes:

[0022] The wavelet transform submodule is used to perform multi-level wavelet transform on the input data of the current spectral feature extraction and fusion module to obtain N frequency components corresponding to N variables;

[0023] Perform a dimension alignment operation on N frequency components to obtain N frequency components after dimension alignment;

[0024] The N frequency components after alignment are aggregated to obtain the wavelet-transformed data.

[0025] Preferably, the N frequency components are obtained by the following formula:

[0026]

[0027] The N frequency components after alignment are obtained using the following formula:

[0028]

[0029] The wavelet-transformed data is obtained using the following formula:

[0030]

[0031] In the formula, This represents the input data for the l-th spectral feature extraction and fusion module, where Dwt(·) represents the multi-level wavelet transform, and f1, f2, ..., f N These represent the 1st, 2nd, ..., Nth frequency components, respectively. These represent the 1st, 2nd, ..., Nth frequency components after alignment, respectively. Padding(·) represents the alignment dimension operation. This represents the data after wavelet transform, and Strack(·) represents the aggregation operation.

[0032] Preferably, a multi-scale parallel 2D convolutional network is used to extract features from the wavelet transform data, resulting in feature-extracted data including:

[0033] The wavelet-transformed data are input into the six convolutional layers of one of the parallel convolutional sub-networks;

[0034] The outputs of the six convolutional layers of one of the parallel convolutional sub-networks are concatenated, the average of the concatenated results is obtained, and the average of the concatenated results is then input into the six convolutional layers of another parallel convolutional sub-network.

[0035] The outputs of the six convolutional layers of another parallel convolutional sub-network are concatenated, and the mean of the concatenated result is obtained. This mean is then used as the data after feature extraction.

[0036] The six convolutional layers employ dilated convolution with kernel sizes of 1, 3, 5, 7, 9, and 11, and dilation rates of 0, 1, 2, 3, 4, and 5, respectively.

[0037] Preferably, the data after feature extraction is obtained using the following formula:

[0038]

[0039] In the formula, Conv_2D h (·) represents the h-th convolutional operation in a multi-scale parallel 2D convolutional network, where H = 6 represents the number of convolutional layers, Ser[·], Mean(·). i Stack(·) represents the serial operation, the averaging operation of the i-th parallel convolutional subnetwork, and the tensor concatenation operation, respectively. I = 2 represents the number of parallel convolutional subnetworks. This represents the data after feature extraction.

[0040] Preferably, a sparse self-attention feature fusion module network is used to perform feature fusion on the feature-extracted data to obtain feature-fused data. The feature-fused data is the output data of the current spectral feature extraction and fusion module, including:

[0041] A sparse self-attention feature fusion module network is used to perform linear mapping on the feature-extracted data to obtain key values, query matrix, and value matrix;

[0042] Attention weights are obtained by inputting the key-value and query matrix into the activation function.

[0043] Randomly downsample the attention weights in the frequency dimension to obtain the downsampled attention weights.

[0044] The feature-fused data is obtained based on the value matrix and the attention weights after downsampling.

[0045] Preferably, the attention weight is obtained by the following formula:

[0046] Attent_weight = sigmoid(K·Q) T )

[0047] The fused data is obtained using the following formula:

[0048]

[0049] In the formula, Attent_weight represents the attention weight, sigmoid(·) represents the activation function, K represents the key value, Q represents the query matrix, V represents the value matrix, and Attent_weight′ represents the downsampled attention weight. This represents the data after feature fusion.

[0050] Preferably, the time-domain loss is obtained by the following formula:

[0051] Loss_time = MSE(X″, Y)

[0052] The frequency domain loss is obtained using the following formula:

[0053] Loss_freq=MSE(FFT(X″),FFT(Y))

[0054] The total loss can be obtained using the following formula:

[0055] Loss = Loss_time + σ * Loss_freq

[0056] In the formula, Loss_time represents the time domain loss, MSE(X",Y) represents the root mean square error of X",Y, X" and Y represent the predicted time series and the original time series, respectively, Loss_freq represents the frequency domain loss, FFT(·) represents the Fourier transform, MSE(FFT(X"),FFT(Y)) represents the root mean square error of FFT(X"),FFT(Y), Loss represents the total loss, and σ represents the loss constant, which takes a value of 0 to 1.

[0057] Preferably, the original time series is electricity data, financial data, weather data, or traffic data.

[0058] The technical solution of this invention firstly fuses and encodes time-series data in the time, spatial, and shallow semantic dimensions through a semantic embedding layer. Then, wavelet transform is used to perform time-frequency decomposition and data structure reorganization of the time-series data, mapping the 1D time variation pattern to a 2D time-frequency variation pattern in the wavelet domain. Next, a learnable 2D convolutional kernel is used for efficient joint extraction of time-frequency variation features. Finally, a self-attention mechanism incorporating a random downsampling strategy is employed to further capture long-term dependency features in the time-series data, and high-level semantic feature dimensionality reduction and fusion are performed, thereby achieving high-precision sequence data prediction. The long-term series prediction method based on 2D time-frequency variation modeling described in this invention solves the problem of difficult feature extraction caused by the rich and varied time patterns in time-series data, improving the long-term prediction performance of time-series data. This invention has significant application value in open-source fields such as weather forecasting, energy and smart grid management, and disease transmission analysis. Attached Figure Description

[0059] The accompanying drawings, which form part of this specification, are provided to further illustrate embodiments of the invention and, together with the textual description, explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0060] Figure 1 A flowchart of a long-time series prediction method based on time-frequency variation pattern modeling according to an embodiment of the present invention is shown;

[0061] Figure 2 A schematic diagram of the structure of a multi-scale parallel 2D convolutional network provided according to an embodiment of the present invention is shown. Detailed Implementation

[0062] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0064] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0065] like Figure 1 and Figure 2 As shown, this invention provides a long-term series prediction method based on time-frequency variation pattern modeling, the method comprising:

[0066] The original time series is feature-encoded to obtain the encoded time series.

[0067] The encoded time series is input into the time series prediction model for training, resulting in the predicted time series output by the model. The time series prediction model comprises L residually connected spectral feature extraction and fusion blocks (SFEF-Blocks). During each training iteration, the input and output data of the current SFEF-Block are fused, and the fusion result is used as the input data for the next SFEF-Block. The input data for the first SFEF-Block is the encoded time series, and the fusion result of the output data and input data of the last SFEF-Block is the predicted time series output by the time series prediction model.

[0068] The time-domain loss of the time series prediction model is obtained based on the original time series and the predicted time series;

[0069] Fourier transforms are performed on the original time series and the predicted time series respectively. The frequency domain loss of the time series prediction model is obtained based on the original time series and the predicted time series after the Fourier transform.

[0070] The total loss of the time series prediction model is obtained based on the time domain loss and the frequency domain loss.

[0071] The structural parameters of the time series prediction model are updated based on the total loss to obtain the updated time series prediction model. The encoded time series is then trained based on the updated time series prediction model until the preset number of training iterations is reached to obtain the trained time series prediction model.

[0072] Collect time series corresponding to the application scenario, encode the features of the time series corresponding to the application scenario, and obtain the encoded time series corresponding to the application scenario.

[0073] Based on the trained time series prediction model, the encoded time series corresponding to the application scenario is predicted to obtain the predicted time series corresponding to the application scenario.

[0074] This invention first fuses and encodes time-series data in the temporal, spatial, and shallow semantic dimensions through a semantic embedding layer. Then, it utilizes wavelet transform to perform time-frequency decomposition and data structure reorganization, mapping 1D time-varying patterns to 2D time-frequency patterns in the wavelet domain. Next, it employs learnable 2D convolutional kernels for efficient joint extraction of time-frequency variation features. Finally, it employs a self-attention mechanism with a random downsampling strategy to further capture long-term dependency features in the time-series data and performs high-level semantic feature dimensionality reduction and fusion, thereby achieving high-precision sequence data prediction. The long-term series prediction method based on 2D time-frequency variation modeling described in this invention solves the problem of difficult feature extraction caused by the rich and varied temporal patterns in time-series data, improving the long-term prediction performance of time-series data. This invention has significant application value in open-source fields such as weather forecasting, energy and smart grid management, and disease transmission analysis.

[0075] According to one embodiment of the present invention, feature encoding of the original time series to obtain the encoded time series includes: feature encoding of the original time series based on a feature encoding network to obtain the encoded time series.

[0076] In this embodiment, the feature encoding network is used to perform high-dimensional mapping on the original time series, and the calculation formula is as follows:

[0077]

[0078] In the formula, X 1D ∈R T×NLet R represent the original time series with a time length of T and N variables, where R represents the real domain, and Embed(·) is a semantic embedding layer operation, which consists of three parts: a position encoding subnetwork, a global time stamp encoding subnetwork, and a scalar projection subnetwork. This represents the encoded time series, where M is the dimension of the encoded features.

[0079] According to an embodiment of the present invention, the fusion result of the input data and output data of the current spectral feature extraction and fusion module is obtained by the following formula:

[0080]

[0081] In the formula, This represents the input data for the l-th spectral feature extraction and fusion module. This represents the output data of the l-th spectral feature extraction and fusion module. This represents the fusion result of the input and output data of the l-th spectral feature extraction and fusion module.

[0082] According to one embodiment of the present invention, each spectral feature extraction and fusion module includes a wavelet transform sub-module, a multi-scale parallel 2D convolutional network and a sparse self-attention feature fusion module network connected in series, and each multi-scale parallel 2D convolutional network includes two parallel convolutional sub-networks connected in series.

[0083] The wavelet transform submodule is used to perform wavelet transform on the input data of the current spectral feature extraction and fusion module to obtain the wavelet-transformed data;

[0084] A multi-scale parallel 2D convolutional network is used to extract features from the wavelet transform data to obtain the feature-extracted data.

[0085] A sparse self-attention feature fusion module network is used to fuse the features extracted from the data to obtain the fused data, which is the output data of the current spectral feature extraction and fusion module.

[0086] For any SFEF-Block, the detailed data processing flow is as follows: the encoded data... Wavelet transform and data structure reorganization operations are performed to construct a 2D semantic representation space. Then, 2D convolutional kernels are used for multi-scale feature extraction, followed by deep feature fusion representation through a sparse self-attention mechanism. This process can be represented as follows:

[0087] The wavelet-transformed data is obtained using the following formula:

[0088]

[0089] The data after feature extraction is obtained using the following formula:

[0090]

[0091] The fused data is obtained using the following formula:

[0092]

[0093] In the formula, This represents the data after wavelet transform, and DWT(·) represents the wavelet transform submodule. The data represents the extracted features. Convs_2D(·) represents a multi-scale parallel 2D convolutional network, and Atten(·) represents a sparse self-attention feature fusion module network. C represents the complex domain, F represents the frequency dimension size, and J represents the number of frequency components.

[0094] According to one embodiment of the present invention, a wavelet transform submodule is used to perform wavelet transform on the input data of the current spectral feature extraction and fusion module to obtain wavelet-transformed data, including:

[0095] The wavelet transform submodule is used to perform multi-level wavelet transform on the input data of the current spectral feature extraction and fusion module to obtain N frequency components corresponding to N variables;

[0096] Perform a dimension alignment operation on N frequency components to obtain N frequency components after dimension alignment;

[0097] The N frequency components after alignment are aggregated to obtain the wavelet-transformed data.

[0098] Specifically, time series data is a highly abstract information body existing in the form of 1D data. However, existing time series data processing often only extracts features in the time domain, making it difficult to uncover the more comprehensive and deeper semantic information of time series data. Therefore, this invention uses wavelet transform (WT) to transform time series data from the time domain to the time-frequency domain, thereby semantically decoupling the time series data in the frequency dimension. Then, the frequency components are concatenated and recombined to construct a 2D semantic representation space for the time series data, which is beneficial for subsequent more efficient multi-dimensional feature extraction using 2D convolution. The calculation process of the wavelet transform module is as follows:

[0099] The N frequency components are obtained using the following formula:

[0100]

[0101] The N frequency components after alignment are obtained using the following formula:

[0102]

[0103] The wavelet-transformed data is obtained using the following formula:

[0104]

[0105] In the formula, This represents the input data for the l-th spectral feature extraction and fusion module, where Dwt(·) represents the multi-level wavelet transform, and f1, f2, ..., f N These represent the 1st, 2nd, ..., Nth frequency components, respectively. These represent the 1st, 2nd, ..., Nth frequency components after alignment, respectively. Padding(·) represents the alignment dimension operation. This represents the data after wavelet transform, and `Strack(·)` represents the aggregation operation. Wherein, It is a 2D matrix.

[0106] According to one embodiment of the present invention, such as Figure 2 As shown, a multi-scale parallel 2D convolutional network is used to extract features from the wavelet transform data. The data after feature extraction includes:

[0107] The wavelet-transformed data are input into the six convolutional layers of one of the parallel convolutional sub-networks;

[0108] The outputs of the six convolutional layers of one of the parallel convolutional sub-networks are concatenated, the average of the concatenated results is obtained, and the average of the concatenated results is then input into the six convolutional layers of another parallel convolutional sub-network.

[0109] The outputs of the six convolutional layers of another parallel convolutional sub-network are concatenated, and the mean of the concatenated result is obtained. This mean is then used as the data after feature extraction.

[0110] The six convolutional layers employ dilated convolution with kernel sizes of 1, 3, 5, 7, 9, and 11, and dilation rates of 0, 1, 2, 3, 4, and 5, respectively.

[0111] In this embodiment, multi-scale convolutional parallel operations are used to capture and extract periodic patterns (global) and abrupt change patterns (local) in time-series data, and then the features are fused and output using mean aggregation. The calculation process is as follows:

[0112] The data after feature extraction is obtained using the following formula:

[0113]

[0114] In the formula, Conv_2D h (·) represents the h-th convolutional operation in a multi-scale parallel 2D convolutional network, where H = 6 represents the number of convolutional layers, Ser[·], Mean(·). i Stack(·) represents the serial operation, the averaging operation of the i-th parallel convolutional subnetwork, and the tensor concatenation operation, respectively. I = 2 represents the number of parallel convolutional subnetworks. This represents the data after feature extraction.

[0115] Using 2D convolution can more efficiently extract multi-dimensional features from wavelet-transformed time-series data, capturing multi-scale time-domain and frequency-domain variation patterns. Furthermore, multi-scale modeling of the relationships between time-series variables is beneficial for capturing long-distance dependencies in the time-series data.

[0116] To further improve feature utilization efficiency and capture deeper temporal and frequency variation patterns, this invention proposes a sparse self-attention mechanism for feature selection and fusion. Specifically, based on the traditional self-attention mechanism, a random sampling strategy is introduced. This approach, on the one hand, further mines long-distance dependency patterns in time-series data, enhancing the efficiency of utilizing deep semantic information; on the other hand, by randomly masking attention weights, it reduces computational complexity and achieves efficient feature selection and fusion representation.

[0117] According to one embodiment of the present invention, a sparse self-attention feature fusion module network is used to perform feature fusion on the feature-extracted data to obtain feature-fused data. The feature-fused data is the output data of the current spectral feature extraction and fusion module, including:

[0118] A sparse self-attention feature fusion module network is used to perform linear mapping on the feature-extracted data to obtain key values, query matrix, and value matrix;

[0119] Attention weights are obtained by inputting the key-value and query matrix into the activation function.

[0120] Randomly downsample the attention weights in the frequency dimension to obtain the downsampled attention weights.

[0121] The feature-fused data is obtained based on the value matrix and the attention weights after downsampling.

[0122] Specifically, the attention weight is obtained through the following formula:

[0123] Attent_weight = sigmoid(K·Q) T )

[0124] The fused data is obtained using the following formula:

[0125]

[0126] In the formula, Attent_weight represents the attention weight, sigmoid(·) represents the activation function, K represents the key value, Q represents the query matrix, V represents the value matrix, and Attent_weight′ represents the downsampled attention weight. This represents the data after feature fusion. Where K, Q, V ∈ R M×W And W = F × Q, Attent_weight ∈ R W×W Attent_weight′∈R W×T ,

[0127] According to one embodiment of the present invention, the time-domain loss is obtained by the following formula:

[0128] Loss_time = MSE(X″, Y)

[0129] The frequency domain loss is obtained using the following formula:

[0130] Loss_freq=MSE(FFT(X″),FFT(Y))

[0131] The total loss can be obtained using the following formula:

[0132] Loss = Loss_time + σ * Loss_freq

[0133] In the formula, Loss_time represents the time domain loss, MSE(X",Y) represents the root mean square error of X",Y, X" and Y represent the predicted time series and the original time series, respectively, Loss_freq represents the frequency domain loss, FFT(·) represents the Fourier transform, MSE(FFT(X"),FFT(Y)) represents the root mean square error of FFT(X"),FFT(Y), Loss represents the total loss, and σ represents the loss constant, which takes a value of 0 to 1.

[0134] According to one embodiment of the present invention, the original time series is electricity data, financial data, weather data, or traffic data. The original time series is part of the data in this document.

[0135] Specifically, power data can be obtained from power transformer temperature data or electricity consumption data; financial data includes daily exchange rate data; weather data includes temperature and humidity; and traffic data includes road occupancy measured by different sensors.

[0136] To gain a further understanding of the present invention, the long-time series prediction method based on time-frequency variation pattern modeling of the present invention will be described in detail below.

[0137] To verify the effectiveness of the method in time series data prediction, long-term series prediction—the most challenging task in time series prediction—was selected as the validation experimental group. The benchmark datasets used included five major open-source datasets: ETT, Electricity, Exchange Rate, Weather, and Traffic, covering real-world application scenarios such as electricity, finance, weather, and transportation. Dataset details are as follows:

[0138] The ETT dataset contains two years of power transformer temperature data collected from two different counties in China. It comprises four sub-datasets based on different adoption frequencies: {ETTh1, ETTh2} at the 1-hour level and {ETTm1, ETTm2} at the 15-minute level. Each data point consists of the target value "oil temperature" and six power load characteristics.

[0139] The Electricity dataset contains hourly electricity consumption data for 321 customers recorded between 2016 and 2019.

[0140] The ExchangeRate dataset records daily exchange rates for eight different countries between 1990 and 2010.

[0141] The Weather dataset records meteorological data throughout 2020, with data recorded every 10 minutes. Each data point contains 21 meteorological indicators, such as temperature and humidity.

[0142] The Traffic dataset contains hourly traffic data collected by the California Department of Transportation between 2016 and 2018, describing road occupancy as measured by different sensors on highways in the San Francisco Bay Area. Each data point contains 861 feature values.

[0143] Each of the above datasets contains a continuous time series data segment, and sample data is obtained through a sliding window method. In the experiment, the length of the input time series is 96, and the length of the predicted time series is [96, 192, 336, 720].

[0144] The comparison methods used in this verification experiment include the most advanced RNN-type models (LSSL (2022)), CNN-type models (TimesNet (2023)), MLP-type models (LightTS (2022), Dlinear (2023)) and Transformer-type models (FEDformer (2022), Informer (2021)), a total of 6 benchmark methods.

[0145] In addition, the algorithm evaluation metrics used in the verification experiment were root mean square error (MSE) and mean absolute error (MAE).

[0146] The results of this verification experiment are shown in Table 1. It can be seen that the time series data prediction algorithm proposed in this invention achieved the best results, which demonstrates the effectiveness of the algorithm proposed in this invention.

[0147] Table 1 shows the experimental results (underlined values ​​indicate optimal results).

[0148]

[0149]

[0150] In summary, this invention provides a long-term series prediction method based on time-frequency variation pattern modeling. First, time-series data is fused and encoded in the time, spatial, and shallow semantic dimensions through a semantic embedding layer. Then, wavelet transform is used to perform time-frequency decomposition and data structure reorganization of the time-series data, mapping the 1D time variation pattern to a 2D time-frequency variation pattern in the wavelet domain. Next, a learnable 2D convolutional kernel is used for efficient joint extraction of time-frequency variation features. Finally, a self-attention mechanism incorporating a random downsampling strategy is employed to further capture long-term dependency features in the time-series data, and high-level semantic feature dimensionality reduction and fusion are performed, thereby achieving high-precision sequence data prediction. The long-term series prediction method based on 2D time-frequency variation modeling described in this invention solves the problem of difficult feature extraction caused by the rich and varied time patterns in time-series data, improving the long-term prediction performance of time-series data. This invention has significant application value in open-source fields such as weather forecasting, energy and smart grid management, and disease transmission analysis.

[0151] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0152] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore should not be construed as limiting the scope of protection of this invention.

[0153] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A long-term series prediction method based on time-frequency variation pattern modeling, characterized in that, The method includes: The original time series is feature-encoded to obtain the encoded time series. The encoded time series is input into a time series prediction model based on time-frequency variation patterns for training, resulting in the predicted time series output by the model. The time series prediction model comprises L residually connected spectral feature extraction and fusion modules. During each training iteration, the input and output data of the current spectral feature extraction and fusion module are fused, and the fusion result is used as the input data for the next module. The input data for the first module is the encoded time series, and the fusion result of the output data and input data of the last module is the predicted time series output by the model. The time-domain loss of the time series prediction model is obtained based on the original time series and the predicted time series; Fourier transforms are performed on the original time series and the predicted time series respectively. The frequency domain loss of the time series prediction model is obtained based on the original time series and the predicted time series after the Fourier transform. The total loss of the time series prediction model is obtained based on the time domain loss and the frequency domain loss. The structural parameters of the time series prediction model are updated based on the total loss to obtain the updated time series prediction model. The encoded time series is then trained based on the updated time series prediction model until the preset number of training iterations is reached to obtain the trained time series prediction model. Collect time series corresponding to the application scenario, encode the features of the time series corresponding to the application scenario, and obtain the encoded time series corresponding to the application scenario. Based on the trained time series prediction model, the encoded time series corresponding to the application scenario is predicted to obtain the predicted time series corresponding to the application scenario. The original time series data includes power data, financial data, weather data, or traffic data. Power data uses power transformer temperature data or electricity consumption data, financial data includes daily exchange rate data, weather data includes temperature and humidity, and traffic data includes road occupancy rates measured by different sensors. Each spectral feature extraction and fusion module includes a wavelet transform sub-module connected in series, a multi-scale parallel 2D convolutional network, and a sparse self-attention feature fusion module network. Each multi-scale parallel 2D convolutional network includes two parallel convolutional sub-networks connected in series. The wavelet transform submodule is used to perform wavelet transform on the input data of the current spectral feature extraction and fusion module to obtain the wavelet-transformed data; A multi-scale parallel 2D convolutional network is used to extract features from the wavelet transform data to obtain the feature-extracted data. A sparse self-attention feature fusion module network is used to fuse the features extracted from the data to obtain the fused data, which is the output data of the current spectral feature extraction and fusion module.

2. The method according to claim 1, characterized in that, The fusion result of the input and output data of the current spectral feature extraction and fusion module is obtained by the following formula: In the formula, This represents the input data for the l-th spectral feature extraction and fusion module. This represents the output data of the l-th spectral feature extraction and fusion module. This represents the fusion result of the input and output data of the l-th spectral feature extraction and fusion module.

3. The method according to claim 1, characterized in that, The wavelet transform submodule is used to perform wavelet transform on the input data of the current spectral feature extraction and fusion module, resulting in the following wavelet-transformed data: The wavelet transform submodule is used to perform multi-level wavelet transform on the input data of the current spectral feature extraction and fusion module to obtain N frequency components corresponding to N variables; Perform a dimension alignment operation on N frequency components to obtain N frequency components after dimension alignment; The N frequency components after alignment are aggregated to obtain the wavelet-transformed data.

4. The method according to claim 3, characterized in that, The N frequency components are obtained using the following formula: The N frequency components after alignment are obtained using the following formula: The wavelet-transformed data is obtained using the following formula: In the formula, This represents the input data for the l-th spectral feature extraction and fusion module, where Dwt(·) represents the multi-level wavelet transform, and f1, f2, ..., f N These represent the 1st, 2nd, ..., Nth frequency components, respectively. These represent the 1st, 2nd, ..., Nth frequency components after alignment, respectively. Padding(·) represents the alignment dimension operation. This represents the data after wavelet transform, and Strack(·) represents the aggregation operation.

5. The method according to claim 1, characterized in that, A multi-scale parallel 2D convolutional network is used to extract features from the wavelet transform data. The resulting data includes: The wavelet-transformed data are input into the six convolutional layers of one of the parallel convolutional sub-networks; The outputs of the six convolutional layers of one of the parallel convolutional sub-networks are concatenated, the average of the concatenated results is obtained, and the average of the concatenated results is then input into the six convolutional layers of another parallel convolutional sub-network. The outputs of the six convolutional layers of another parallel convolutional sub-network are concatenated, and the mean of the concatenated result is obtained. This mean is then used as the data after feature extraction. The six convolutional layers employ dilated convolution with kernel sizes of 1, 3, 5, 7, 9, and 11, and dilation rates of 0, 1, 2, 3, 4, and 5, respectively.

6. The method according to claim 5, characterized in that, The data after feature extraction is obtained using the following formula: In the formula, Conv_2D h (·) represents the h-th convolutional operation in a multi-scale parallel 2D convolutional network, where H = 6 represents the number of convolutional layers, Ser[·], Mean(·). i Stack(·) represents the serial operation, the averaging operation of the i-th parallel convolutional subnetwork, and the tensor concatenation operation, respectively. I = 2 represents the number of parallel convolutional subnetworks. This represents the data after feature extraction.

7. The method according to claim 1, characterized in that, A sparse self-attention feature fusion module network is used to fuse the features extracted from the data, resulting in fused feature data. The fused feature data is the output data of the current spectral feature extraction and fusion module, including: A sparse self-attention feature fusion module network is used to perform linear mapping on the feature-extracted data to obtain key values, query matrix, and value matrix; Attention weights are obtained by inputting the key-value and query matrix into the activation function. Randomly downsample the attention weights in the frequency dimension to obtain the downsampled attention weights. The feature-fused data is obtained based on the value matrix and the attention weights after downsampling.

8. The method according to claim 1, characterized in that, The attention weight is obtained using the following formula: Attent_weight=sigmoid(K·Q T ) The fused data is obtained using the following formula: In the formula, Attent_weight represents the attention weight, sigmoid(·) represents the activation function, K represents the key value, Q represents the query matrix, V represents the value matrix, and Attent_weight′ represents the downsampled attention weight. This represents the data after feature fusion.

9. The method according to claim 1, characterized in that, The time-domain loss is obtained using the following formula: Loss_time = MSE(X″, Y) The frequency domain loss is obtained using the following formula: Loss_freq=MSE(FFT(X″),FFT(Y)) The total loss can be obtained using the following formula: Loss = Loss_time + σ * Loss_freq In the formula, Loss_time represents the time domain loss, MSE(X",Y) represents the root mean square error of X",Y, X" and Y represent the predicted time series and the original time series, respectively, Loss_freq represents the frequency domain loss, FFT(·) represents the Fourier transform, MSE(FFT(X"),FFT(Y)) represents the root mean square error of FFT(X"),FFT(Y), Loss represents the total loss, and σ represents the loss constant, which takes a value of 0 to 1.

Images