Long-time sequence prediction method based on time-frequency change mode modeling

Abstract

The invention provides a long time sequence prediction method based on time-frequency change mode modeling, and the method comprises the steps: carrying out the feature coding of an original time sequence, and obtaining a coded time sequence; inputting the coded time sequence into a time sequence prediction model for training to obtain a predicted time sequence output by the time sequence prediction model; obtaining the total loss of the time sequence prediction model based on the time domain loss and the frequency domain loss; updating structure parameters of the time sequence prediction model based on the total loss to obtain an updated time sequence prediction model, and continuing to train the encoded time sequence based on the updated time sequence prediction model until a preset training frequency is reached to obtain a trained time sequence prediction model; obtaining a coded time sequence corresponding to the application scene; and obtaining a prediction time sequence corresponding to the application scene.

Description

Technical Field

[0001] The present invention relates to the technical field of time series data modeling, and in particular to a long time series prediction method based on time-frequency variation pattern modeling. Background Art

[0002] Time series data refers to data recorded by observing certain things or phenomena at a certain sampling frequency within a certain time range. Time series data prediction is to capture long-term dependency features for modeling in order to predict future development trends. At present, the use of deep learning technology for time series data prediction is the mainstream of research. Therefore, many advanced methods have emerged. For example, the method based on recurrent neural network has very flexible nonlinear modeling capabilities, but it is easy to have gradient vanishing and explosion problems under long time steps. The convolutional neural network method is limited to the 1D convolution feature extraction method, and can only model data between adjacent time points, and cannot solve the long-term dependency problem. The transformer-based method uses the attention mechanism for feature extraction and time series relationship modeling. However, time series data has abstract deep dependencies, and it is very difficult to model stable and reliable dependencies from discrete time points.

[0003] Therefore, the rich and complex temporal variation patterns contained in time series data severely limit the existing methods for fine-grained semantic analysis. Summary of the invention

[0004] The present invention provides a long time series prediction method based on time-frequency change pattern modeling, which can solve the technical problem that the time series data is poorly predicted due to the rich and complex time change patterns coupled in the time series data.

[0005] The present invention provides a long time series prediction method based on time-frequency variation pattern modeling, the method comprising:

[0006] Perform feature encoding on the original time series to obtain the encoded time series;

[0007] The encoded time series is input into a time series prediction model based on time-frequency variation pattern modeling for training, and a predicted time series output by the time series prediction model is obtained; wherein the time series prediction model includes L residual-connected spectrum feature extraction and fusion modules; during each training, the input data and output data of the current spectrum feature extraction and fusion module are fused, and the fusion result is used as the input data of the next spectrum feature extraction and fusion module; the input data of the first spectrum feature extraction and fusion module is the encoded time series, and the fusion result of the output data of the last spectrum feature extraction and fusion module and the input data of the last spectrum feature extraction and fusion module is the predicted time series output by the time series prediction model;

[0008] The time domain loss of the time series prediction model is obtained based on the original time series and the predicted time series;

[0009] Fourier transform is performed on the original time series and the predicted time series respectively, and the frequency domain loss of the time series prediction model is obtained based on the original time series and the predicted time series after Fourier transform;

[0010] The total loss of the time series prediction model is obtained based on the time domain loss and frequency domain loss;

[0011] Based on the total loss, the structural parameters of the time series prediction model are updated to obtain an updated time series prediction model, and the encoded time series is continuously trained based on the updated time series prediction model until a preset number of training times is reached to obtain a trained time series prediction model;

[0012] Collecting a time series corresponding to the application scenario, performing feature encoding on the time series corresponding to the application scenario, and obtaining an encoded time series corresponding to the application scenario;

[0013] Based on the trained time series prediction model, the encoded time series corresponding to the application scenario is predicted to obtain a predicted time series corresponding to the application scenario.

[0014] Preferably, the fusion result of the input data and output data of the current spectrum feature extraction and fusion module is obtained by the following formula:

[0015]

[0016] In the formula, represents the input data of the lth spectrum feature extraction and fusion module, represents the output data of the lth spectrum feature extraction and fusion module, Represents the fusion result of the input data and output data of the lth spectrum feature extraction and fusion module.

[0017] Preferably, each spectrum feature extraction and fusion module includes a wavelet transform submodule, a multi-scale parallel 2D convolutional network and a sparse self-attention feature fusion module network connected in series, and each multi-scale parallel 2D convolutional network includes two parallel convolutional subnetworks connected in series;

[0018] The wavelet transform submodule is used to perform wavelet transform on the input data of the current spectrum feature extraction and fusion module to obtain the wavelet transformed data;

[0019] A multi-scale parallel 2D convolutional network is used to extract features from the wavelet transformed data to obtain feature-extracted data;

[0020] A sparse self-attention feature fusion module network is used to perform feature fusion on the feature-extracted data to obtain feature-fused data, which is the output data of the current spectrum feature extraction and fusion module.

[0021] Preferably, the wavelet transform submodule is used to perform wavelet transform on the input data of the current spectrum feature extraction and fusion module, and the data obtained after wavelet transform includes:

[0022] The wavelet transform submodule is used to perform multi-layer wavelet transform on the input data of the current spectrum feature extraction and fusion module to obtain N frequency components corresponding to N variables;

[0023] Performing dimension alignment operation on the N frequency components to obtain N frequency components after dimension alignment;

[0024] Aggregate the N frequency components after dimensional alignment to obtain wavelet transformed data.

[0025] Preferably, N frequency components are obtained by the following formula:

[0026]

[0027] The N frequency components after the alignment dimension are obtained by the following formula:

[0028]

[0029] The data after wavelet transformation is obtained by the following formula:

[0030]

[0031] In the formula, represents the input data of the lth spectrum feature extraction and fusion module, Dwt(·) represents multi-layer wavelet transform, f1,f2,…,f N Represent the 1st, 2nd, ..., Nth frequency components respectively, They represent the 1st, 2nd, ..., Nth frequency components after the alignment dimension, respectively. Padding(·) represents the alignment dimension operation. represents the data after wavelet transformation, and Strack(·) represents the aggregation operation.

[0032] Preferably, a multi-scale parallel 2D convolutional network is used to extract features from the wavelet transformed data, and the feature-extracted data includes:

[0033] The wavelet transformed data are input into the 6 convolutional layers of one of the parallel convolutional sub-networks;

[0034] The output results of the six convolutional layers of one of the parallel convolutional sub-networks are concatenated to obtain the mean of the concatenated results, and the mean of the concatenated results is input into the six convolutional layers of another parallel convolutional sub-network respectively.

[0035] The output results of the six convolutional layers of another parallel convolutional subnetwork are concatenated to obtain the mean of the concatenated results, and the mean of the concatenated results is used as the data after feature extraction;

[0036] Among them, 6 convolutional layers use dilated convolution, and the convolution kernel sizes are 1, 3, 5, 7, 9, and 11, and the dilation rates are 0, 1, 2, 3, 4, and 5, respectively.

[0037] Preferably, the data after feature extraction is obtained by the following formula:

[0038]

[0039] In the formula, Conv_2D h (·) represents the hth convolution operation of the multi-scale parallel 2D convolutional network, H = 6 represents the number of convolutional layers, Ser[·], Mean(·) i , Stack(·) represents the serial operation, the averaging operation of the i-th parallel convolutional subnetwork, and the tensor concatenation operation, respectively. I = 2 represents the number of parallel convolutional subnetworks. Represents the data after feature extraction.

[0040] Preferably, a sparse self-attention feature fusion module network is used to perform feature fusion on the feature-extracted data to obtain feature-fused data, and the feature-fused data is the output data of the current spectrum feature extraction and fusion module, including:

[0041] A sparse self-attention feature fusion module network is used to linearly map the feature-extracted data to obtain key values, query matrices, and value matrices.

[0042] Based on the key value and query matrix, input them into the activation function to get the attention weight;

[0043] Randomly downsample the attention weights in the frequency dimension to obtain the downsampled attention weights;

[0044] The feature fused data is obtained based on the value matrix and the downsampled attention weights.

[0045] Preferably, the attention weight is obtained by the following formula:

[0046] Attent_weight = sigmoid(K·Q T )

[0047] The data after feature fusion is obtained through the following formula:

[0048]

[0049] Where Attent_weight represents the attention weight, sigmoid(·) represents the activation function, K represents the key value, Q represents the query matrix, V represents the value matrix, Attent_weight′ represents the attention weight after downsampling, Represents the data after feature fusion.

[0050] Preferably, the time domain loss is obtained by the following formula:

[0051] Loss_time=MSE(X″,Y)

[0052] The frequency domain loss is obtained by the following formula:

[0053] Loss_freq=MSE(FFT(X″),FFT(Y))

[0054] The total loss is obtained by the following formula:

[0055] Loss = Loss_time + σ * Loss_freq

[0056] Where Loss_time represents the time domain loss, MSE(X",Y) represents the root mean square error of X",Y, X", Y represent the predicted time series and the original time series respectively, Loss_freq represents the frequency domain loss, FFT(·) represents Fourier transform, MSE(FFT(X"),FFT(Y)) represents the root mean square error of FFT(X"),FFT(Y), Loss represents the total loss, and σ represents the loss constant, which ranges from 0 to 1.

[0057] Preferably, the original time series is power data, financial data, weather data or traffic data.

[0058] The technical solution of the present invention is applied. First, the time series data is fused and encoded in the time dimension, space dimension and shallow semantic dimension through the semantic embedding layer. Then, the time series data is decomposed into time and frequency and the data structure is reorganized by wavelet transform, and the 1D time change pattern is mapped to the 2D time-frequency change pattern in the wavelet domain. Then, a learnable 2D convolution kernel is used to efficiently extract the time-frequency change features. Finally, a self-attention mechanism that introduces a random downsampling strategy is used to further capture the long-term dependency relationship features in the time series data, and high-level semantic features are reduced and fused, thereby achieving high-precision sequence data prediction. The long-term series prediction method based on 2D time-frequency change modeling described in the present invention solves the problem that the rich and changeable time patterns in the time series data make feature extraction difficult, and improves the long-term prediction performance of time series data. The present invention has great application value in open source fields such as weather forecasting, energy and smart grid management, and disease transmission analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The included drawings are used to provide a further understanding of the embodiments of the present invention, which constitute a part of the specification, are used to illustrate the embodiments of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0060] Figure 1 A flowchart of a long time series prediction method based on time-frequency variation pattern modeling provided according to an embodiment of the present invention is shown;

[0061] Figure 2 A schematic diagram of the structure of a multi-scale parallel 2D convolutional network provided according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0062] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0064] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, numerical expressions and numerical values ​​do not limit the scope of the present invention. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be regarded as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0065] like Figure 1 and Figure 2 As shown, the present invention provides a long time series prediction method based on time-frequency change pattern modeling, the method comprising:

[0066] Perform feature encoding on the original time series to obtain the encoded time series;

[0067] The encoded time series is input into the time series prediction model for training to obtain the predicted time series output by the time series prediction model; wherein the time series prediction model includes L residual-connected spectrum feature extraction and fusion blocks (Spectrum feature extraction and fusion block, SFEF-Block); during each training, the input data and output data of the current spectrum feature extraction and fusion module are fused, and the fusion result is used as the input data of the next spectrum feature extraction and fusion module; the input data of the first spectrum feature extraction and fusion module is the encoded time series, and the fusion result of the output data of the last spectrum feature extraction and fusion module and the input data of the last spectrum feature extraction and fusion module is the predicted time series output by the time series prediction model;

[0068] The time domain loss of the time series prediction model is obtained based on the original time series and the predicted time series;

[0069] Fourier transform is performed on the original time series and the predicted time series respectively, and the frequency domain loss of the time series prediction model is obtained based on the original time series and the predicted time series after Fourier transform;

[0070] The total loss of the time series prediction model is obtained based on the time domain loss and frequency domain loss;

[0071] Based on the total loss, the structural parameters of the time series prediction model are updated to obtain an updated time series prediction model, and the encoded time series is continuously trained based on the updated time series prediction model until a preset number of training times is reached to obtain a trained time series prediction model;

[0072] Collecting a time series corresponding to the application scenario, performing feature encoding on the time series corresponding to the application scenario, and obtaining an encoded time series corresponding to the application scenario;

[0073] Based on the trained time series prediction model, the encoded time series corresponding to the application scenario is predicted to obtain a predicted time series corresponding to the application scenario.

[0074] The present invention firstly performs fusion encoding of time series data in time dimension, space dimension and shallow semantic dimension through semantic embedding layer, then uses wavelet transform to perform time-frequency decomposition and data structure reorganization on time series data, maps 1D time variation pattern to 2D time-frequency variation pattern in wavelet domain, then uses learnable 2D convolution kernel to perform efficient joint extraction of time-frequency variation features, and finally, adopts self-attention mechanism with random downsampling strategy to further capture long-term dependency relationship features in time series data, and performs dimension reduction and fusion of high-level semantic features, so as to achieve high-precision sequence data prediction. The long time series prediction method based on 2D time-frequency variation modeling described in the present invention solves the problem that the rich and changeable time patterns in time series data lead to difficulty in feature extraction, and improves the long-term prediction performance of time series data. The present invention has great application value in open source fields such as weather forecasting, energy and smart grid management, and disease transmission analysis.

[0075] According to an embodiment of the present invention, feature encoding the original time series to obtain the encoded time series includes: feature encoding the original time series based on a feature encoding network to obtain the encoded time series.

[0076] In this embodiment, the feature encoding network is used to perform high-dimensional mapping on the original time series, and the calculation formula is as follows:

[0077]

[0078] Where, X 1D ∈R T×Nrepresents the original time series with a time length of T and N variables, R represents the real domain, and Embed(·) is the semantic embedding layer operation, which consists of three parts: position encoding (Position Embeddings) subnetwork, global timestamp encoding (Global Time Stamp Embeddings) subnetwork and scalar projection (Scalar Projection) subnetwork. represents the encoded time series, and M is the encoded feature dimension.

[0079] According to an embodiment of the present invention, the fusion result of the input data and the output data of the current spectrum feature extraction and fusion module is obtained by the following formula:

[0080]

[0081] In the formula, represents the input data of the lth spectrum feature extraction and fusion module, represents the output data of the lth spectrum feature extraction and fusion module, represents the fusion result of the input data and output data of the lth spectrum feature extraction and fusion module.

[0082] According to an embodiment of the present invention, each spectrum feature extraction and fusion module includes a wavelet transform submodule, a multi-scale parallel 2D convolutional network and a sparse self-attention feature fusion module network connected in series, and each multi-scale parallel 2D convolutional network includes two parallel convolutional subnetworks connected in series;

[0083] The wavelet transform submodule is used to perform wavelet transform on the input data of the current spectrum feature extraction and fusion module to obtain the wavelet transformed data;

[0084] A multi-scale parallel 2D convolutional network is used to extract features from the wavelet transformed data to obtain feature-extracted data;

[0085] A sparse self-attention feature fusion module network is used to perform feature fusion on the feature-extracted data to obtain feature-fused data, which is the output data of the current spectrum feature extraction and fusion module.

[0086] For any SFEF-Block, the detailed data processing flow is as follows: Wavelet transform and data structure reorganization operations are performed to construct a 2D semantic representation space, and then a 2D convolution kernel is used to extract multi-scale features. The sparse self-attention mechanism is used to perform deep feature fusion representation. The process can be expressed as follows:

[0087] The data after wavelet transformation is obtained by the following formula:

[0088]

[0089] The data after feature extraction is obtained by the following formula:

[0090]

[0091] The data after feature fusion is obtained through the following formula:

[0092]

[0093] In the formula, represents the data after wavelet transformation, DWT(·) represents the wavelet transformation submodule, represents the data after feature extraction, Convs_2D(·) represents the multi-scale parallel 2D convolution network, and Atten(·) represents the sparse self-attention feature fusion module network. C represents the complex domain, F represents the frequency dimension, and J represents the number of frequency components.

[0094] According to an embodiment of the present invention, a wavelet transform submodule is used to perform wavelet transform on the input data of the current spectrum feature extraction and fusion module, and the data obtained after wavelet transform includes:

[0095] The wavelet transform submodule is used to perform multi-layer wavelet transform on the input data of the current spectrum feature extraction and fusion module to obtain N frequency components corresponding to N variables;

[0096] Performing dimension alignment operation on the N frequency components to obtain N frequency components after dimension alignment;

[0097] Aggregate the N frequency components after dimensional alignment to obtain wavelet transformed data.

[0098] Specifically, time series data is a highly abstract information body in the form of 1D data, and the existing time series data processing process often only performs feature extraction in the time domain, which makes it difficult to mine more comprehensive and in-depth semantic information of the time series data. Therefore, the present invention uses wavelet transform (WT) to transform the time series data from the time domain to the time-frequency domain, thereby semantically decoupling the time series data in the frequency dimension, and then splicing and reorganizing the frequency components to construct a 2D semantic representation space for the time series data, which is conducive to the subsequent use of 2D convolution for more efficient multi-dimensional feature joint extraction. The calculation process of the wavelet transform module is expressed as follows:

[0099] The N frequency components are obtained by the following formula:

[0100]

[0101] The N frequency components after the alignment dimension are obtained by the following formula:

[0102]

[0103] The data after wavelet transformation is obtained by the following formula:

[0104]

[0105] In the formula, represents the input data of the lth spectrum feature extraction and fusion module, Dwt(·) represents multi-layer wavelet transform, f1,f2,…,f N Represent the 1st, 2nd, ..., Nth frequency components respectively, They represent the 1st, 2nd, ..., Nth frequency components after the alignment dimension, respectively. Padding(·) represents the alignment dimension operation. represents the data after wavelet transformation, and Strack(·) represents the aggregation operation. is a 2D matrix.

[0106] According to an embodiment of the present invention, Figure 2 As shown, a multi-scale parallel 2D convolutional network is used to extract features from the wavelet transformed data, and the data after feature extraction includes:

[0107] The wavelet transformed data are input into the 6 convolutional layers of one of the parallel convolutional sub-networks;

[0108] The output results of the six convolutional layers of one of the parallel convolutional sub-networks are concatenated to obtain the mean of the concatenated results, and the mean of the concatenated results is input into the six convolutional layers of another parallel convolutional sub-network respectively.

[0109] The output results of the six convolutional layers of another parallel convolutional subnetwork are concatenated to obtain the mean of the concatenated results, and the mean of the concatenated results is used as the data after feature extraction;

[0110] Among them, 6 convolutional layers use dilated convolution, and the convolution kernel sizes are 1, 3, 5, 7, 9, and 11, and the dilation rates are 0, 1, 2, 3, 4, and 5, respectively.

[0111] In this embodiment, the periodic pattern (global) and mutation pattern (local) in the time series data are captured and extracted through multi-scale convolution parallel operation, and then the feature fusion output is performed through mean aggregation. The calculation process is shown as follows:

[0112] The data after feature extraction is obtained by the following formula:

[0113]

[0114] In the formula, Conv_2D h (·) represents the hth convolution operation of the multi-scale parallel 2D convolutional network, H = 6 represents the number of convolutional layers, Ser[·], Mean(·) i , Stack(·) represents the serial operation, the averaging operation of the i-th parallel convolutional subnetwork, and the tensor concatenation operation, respectively. I = 2 represents the number of parallel convolutional subnetworks. Represents the data after feature extraction.

[0115] The use of 2D convolution can more efficiently extract multi-dimensional features from time series data after wavelet transformation and capture multi-scale time and frequency domain change patterns. In addition, multi-scale modeling of the correlation between time series variables is also conducive to further capturing the long-distance dependency of time series data.

[0116] In order to further improve the efficiency of feature utilization and capture deeper time-frequency change patterns, the present invention proposes a sparse self-attention mechanism for feature screening and fusion. In detail, based on the traditional self-attention mechanism, a random sampling strategy is introduced. On the one hand, it will further mine the long-distance dependency pattern of time series data and enhance the efficiency of deep semantic information utilization. On the other hand, by randomly shielding the attention weights, the computational complexity is reduced, and efficient feature optimization and fusion representation are achieved.

[0117] According to an embodiment of the present invention, a sparse self-attention feature fusion module network is used to perform feature fusion on the feature-extracted data to obtain feature-fused data, and the feature-fused data is the output data of the current spectrum feature extraction and fusion module, including:

[0118] A sparse self-attention feature fusion module network is used to linearly map the feature-extracted data to obtain key values, query matrices, and value matrices.

[0119] Based on the key value and query matrix, input them into the activation function to get the attention weight;

[0120] Randomly downsample the attention weights in the frequency dimension to obtain the downsampled attention weights;

[0121] The feature fused data is obtained based on the value matrix and the downsampled attention weights.

[0122] Specifically, the attention weight is obtained by the following formula:

[0123] Attent_weight = sigmoid(K·Q T )

[0124] The data after feature fusion is obtained through the following formula:

[0125]

[0126] Where Attent_weight represents the attention weight, sigmoid(·) represents the activation function, K represents the key value, Q represents the query matrix, V represents the value matrix, Attent_weight′ represents the attention weight after downsampling, Represents the data after feature fusion. Among them, K, Q, V∈R M×W , and W = F × Q, Attent_weight ∈ R W×W ,Attent_weight′∈R W×T ,

[0127] According to an embodiment of the present invention, the time domain loss is obtained by the following formula:

[0128] Loss_time=MSE(X″,Y)

[0129] The frequency domain loss is obtained by the following formula:

[0130] Loss_freq=MSE(FFT(X″),FFT(Y))

[0131] The total loss is obtained by the following formula:

[0132] Loss = Loss_time + σ * Loss_freq

[0133] Where Loss_time represents the time domain loss, MSE(X",Y) represents the root mean square error of X",Y, X", Y represent the predicted time series and the original time series respectively, Loss_freq represents the frequency domain loss, FFT(·) represents Fourier transform, MSE(FFT(X"),FFT(Y)) represents the root mean square error of FFT(X"),FFT(Y), Loss represents the total loss, and σ represents the loss constant, which ranges from 0 to 1.

[0134] According to an embodiment of the present invention, the original time series is power data, financial data, weather data or traffic data. The original time series belongs to the data in this article.

[0135] Specifically, power data may include power transformer temperature data or power consumption data, financial data may include daily exchange rate data, weather data may include temperature, humidity, etc., and traffic data may include road occupancy rates measured by different sensors.

[0136] In order to have a further understanding of the present invention, the long time series prediction method based on time-frequency variation pattern modeling of the present invention is described in detail below.

[0137] In order to verify the effectiveness of the proposed method in time series data prediction, the most challenging task in time series prediction tasks, long time series prediction, is selected as the verification experiment group. The benchmark experimental datasets used include five mainstream open source datasets: ETT, electricity, exchangeRate, weather, and traffic, covering real-world application scenarios such as electricity, finance, weather, and transportation. The details of the datasets are as follows:

[0138] The ETT dataset contains two years of power transformer temperature data collected from two different counties in China. According to different adoption frequencies, it contains a total of four sub-datasets, namely {ETTh1, ETTh2} at the 1-hour level and {ETTm1, ETTm2} at the 15-minute level. Each data point consists of a target value "oil temperature" and 6 power load features.

[0139] The Electricity dataset contains hourly electricity usage of 321 customers recorded between 2016 and 2019.

[0140] The ExchangeRate dataset records daily exchange rates for eight different countries between 1990 and 2010.

[0141] The Weather dataset records meteorological data for the entire year of 2020, recorded every 10 minutes, and each data point contains 21 meteorological indicators, such as temperature, humidity, etc.

[0142] The Traffic dataset is traffic data collected hourly by the California Department of Transportation from 2016 to 2018. It describes the road occupancy measured by different sensors on highways in the San Francisco Bay Area, and each data point contains 861 feature values.

[0143] Each of the above datasets contains a continuous time series data, and the sample data is obtained by sliding window. In the experiment, the length of the input time series is 96, and the length of the predicted time series is [96, 192, 336, 720].

[0144] The comparison methods used in this verification experiment include the most cutting-edge RNN models (LSSL (2022)), CNN models (TimesNet (2023)), MLP models (LightTS (2022), Dlinear (2023)) and Transformer models (FEDformer (2022), Informer (2021)), totaling 6 benchmark methods.

[0145] In addition, the algorithm evaluation indicators used in the verification experiment are root mean square error (MSE) and mean absolute error (MAE).

[0146] The results of this verification experiment are shown in Table 1, from which it can be seen that the time series data prediction algorithm proposed in the present invention has achieved the best results, which illustrates the effectiveness of the algorithm proposed in the present invention.

[0147] Table 1 Verification experiment results (underlined indicates optimal)

[0148]

[0149]

[0150] In summary, the present invention provides a long time series prediction method based on time-frequency change pattern modeling. First, the time series data is fused and encoded in the time dimension, space dimension and shallow semantic dimension through the semantic embedding layer. Then, the time series data is decomposed and reorganized in time and frequency by wavelet transform, and the 1D time change pattern is mapped to the 2D time-frequency change pattern in the wavelet domain. Then, a learnable 2D convolution kernel is used to efficiently extract the time-frequency change features. Finally, a self-attention mechanism that introduces a random downsampling strategy is used to further capture the long-term dependency relationship features in the time series data, and high-level semantic features are reduced and fused, thereby achieving high-precision sequence data prediction. The long time series prediction method based on 2D time-frequency change modeling described in the present invention solves the problem that the rich and changeable time patterns in the time series data cause difficulty in feature extraction, and improves the long-term prediction performance of time series data. The present invention has great application value in open source fields such as weather forecasting, energy and smart grid management, and disease transmission analysis.

[0151] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used here to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0152] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.

[0153] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A long time series prediction method based on time-frequency variation pattern modeling, characterized in that: The method comprises: Perform feature encoding on the original time series to obtain the encoded time series; The encoded time series is input into a time series prediction model based on time-frequency variation pattern modeling for training, and a predicted time series output by the time series prediction model is obtained; wherein the time series prediction model includes L residual-connected spectrum feature extraction and fusion modules; during each training, the input data and output data of the current spectrum feature extraction and fusion module are fused, and the fusion result is used as the input data of the next spectrum feature extraction and fusion module; the input data of the first spectrum feature extraction and fusion module is the encoded time series, and the fusion result of the output data of the last spectrum feature extraction and fusion module and the input data of the last spectrum feature extraction and fusion module is the predicted time series output by the time series prediction model; The time domain loss of the time series prediction model is obtained based on the original time series and the predicted time series; Fourier transform is performed on the original time series and the predicted time series respectively, and the frequency domain loss of the time series prediction model is obtained based on the original time series and the predicted time series after Fourier transform; The total loss of the time series prediction model is obtained based on the time domain loss and frequency domain loss; Based on the total loss, the structural parameters of the time series prediction model are updated to obtain an updated time series prediction model, and the encoded time series is continuously trained based on the updated time series prediction model until a preset number of training times is reached to obtain a trained time series prediction model; Collecting a time series corresponding to the application scenario, performing feature encoding on the time series corresponding to the application scenario, and obtaining an encoded time series corresponding to the application scenario; Based on the trained time series prediction model, the encoded time series corresponding to the application scenario is predicted to obtain a predicted time series corresponding to the application scenario.

2. The method according to claim 1, characterized in that The fusion result of the input data and output data of the current spectrum feature extraction and fusion module is obtained by the following formula: In the formula, represents the input data of the lth spectrum feature extraction and fusion module, represents the output data of the lth spectrum feature extraction and fusion module, Represents the fusion result of the input data and output data of the lth spectrum feature extraction and fusion module.

3. The method according to claim 1, characterized in that Each spectrum feature extraction and fusion module includes a wavelet transform submodule, a multi-scale parallel 2D convolutional network, and a sparse self-attention feature fusion module network connected in series, and each multi-scale parallel 2D convolutional network includes two parallel convolutional subnetworks connected in series; The wavelet transform submodule is used to perform wavelet transform on the input data of the current spectrum feature extraction and fusion module to obtain the wavelet transformed data; A multi-scale parallel 2D convolutional network is used to extract features from the wavelet transformed data to obtain feature-extracted data; A sparse self-attention feature fusion module network is used to perform feature fusion on the feature-extracted data to obtain feature-fused data, which is the output data of the current spectrum feature extraction and fusion module.

4. The method according to claim 1, characterized in that The wavelet transform submodule is used to perform wavelet transform on the input data of the current spectrum feature extraction and fusion module, and the data obtained after wavelet transform includes: The wavelet transform submodule is used to perform multi-layer wavelet transform on the input data of the current spectrum feature extraction and fusion module to obtain N frequency components corresponding to N variables; Performing dimension alignment operation on the N frequency components to obtain N frequency components after dimension alignment; Aggregate the N frequency components after dimensional alignment to obtain wavelet transformed data.

5. The method according to claim 4, characterized in that The N frequency components are obtained by the following formula: The N frequency components after the alignment dimension are obtained by the following formula: The data after wavelet transformation is obtained by the following formula: In the formula, represents the input data of the lth spectrum feature extraction and fusion module, Dwt(·) represents multi-layer wavelet transform, f1,f2,…,f N Represent the 1st, 2nd, ..., Nth frequency components respectively, They represent the 1st, 2nd, ..., Nth frequency components after the alignment dimension, respectively. Padding(·) represents the alignment dimension operation. represents the data after wavelet transformation, and Strack(·) represents the aggregation operation.

6. The method according to claim 1, characterized in that A multi-scale parallel 2D convolutional network is used to extract features from the wavelet transformed data. The data after feature extraction includes: The wavelet transformed data are input into the 6 convolutional layers of one of the parallel convolutional sub-networks; The output results of the six convolutional layers of one of the parallel convolutional sub-networks are concatenated to obtain the mean of the concatenated results, and the mean of the concatenated results is input into the six convolutional layers of another parallel convolutional sub-network respectively. The output results of the six convolutional layers of another parallel convolutional subnetwork are concatenated to obtain the mean of the concatenated results, and the mean of the concatenated results is used as the data after feature extraction; Among them, 6 convolutional layers use dilated convolution, and the convolution kernel sizes are 1, 3, 5, 7, 9, and 11, and the dilation rates are 0, 1, 2, 3, 4, and 5, respectively.

7. The method according to claim 6, characterized in that The data after feature extraction is obtained by the following formula: In the formula, Conv_2D h (·) represents the hth convolution operation of the multi-scale parallel 2D convolutional network, H==6 represents the number of convolutional layers, Ser[·], Mean(·)i, Stack(·) represent the serial operation, the mean operation of the i-th parallel convolutional subnetwork, and the tensor concatenation operation, respectively, I==2 represents the number of parallel convolutional subnetworks, Represents the data after feature extraction.

8. The method according to claim 1, characterized in that: The sparse self-attention feature fusion module network is used to fuse the feature extracted data to obtain the feature fused data. The feature fused data is the output data of the current spectrum feature extraction and fusion module, including: A sparse self-attention feature fusion module network is used to linearly map the feature-extracted data to obtain key values, query matrices, and value matrices. Based on the key value and query matrix, input them into the activation function to get the attention weight; Randomly downsample the attention weights in the frequency dimension to obtain the downsampled attention weights; The feature fused data is obtained based on the value matrix and the downsampled attention weights.

9. The method according to claim 1, characterized in that: The attention weight is obtained by the following formula: Attent_weight=sigmoid(K·Q T ) The data after feature fusion is obtained through the following formula: Where Attent_weight represents the attention weight, sigmoid(·) represents the activation function, K represents the key value, Q represents the query matrix, V represents the value matrix, Attent_weight′ represents the attention weight after downsampling, Represents the data after feature fusion.

10. The method according to claim 1, characterized in that The time domain loss is obtained by the following formula: Loss_time = MSE(X, Y) The frequency domain loss is obtained by the following formula: Loss_freq=MSE(FFT(X″),FFT(Y)) The total loss is obtained by the following formula: Loss = Loss_time + σ * Loss_freq Wherein, Loss_time represents the time domain loss, MSE(X″, Y) represents the root mean square error of X″ and Y, X″ and Y represent the predicted time series and the original time series respectively, Loss_freq represents the frequency domain loss, FFT(·) represents Fourier transform, MSE(FFT(X″), FFT(Y)) represents the root mean square error of FFT(X″) and FFT(Y), Loss represents the total loss, and σ represents the loss constant, which ranges from 0 to 1.

11. The method according to claim 1, characterized in that The original time series is electricity data, financial data, weather data, or traffic data.

Images