Multi-scale linear general characterization method for multivariable time series
Through the ALL-MLP architecture and Fourier transform technology, the complex features and universality problems of multivariate time series are solved, efficient and general representation learning is achieved, the model structure is simplified, and it is suitable for multi-task scenarios.
Patent Information
- Application Number
- CN202411090702.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-03
AI Technical Summary
The existing multivariate time series schema model is difficult to accurately capture complex features and provide general representational features, cannot adapt to multitasking scenarios, and has high computational cost and high model complexity.
Using ALL-MLP's lightweight architecture, combining the fast Fourier transform and the '1+M' structure full multi-layer perceptron network, a multi-scale selection strategy and a linear aggregation network are designed, multi-scale spatiotemporal features are extracted, and a general representation is generated.
It realizes high speed, high precision and high versatility characterization learning, simplifies the model structure, reduces the computational cost, is suitable for multi-task scenarios, and generates a more comprehensive feature representation.
Smart Images

Figure CN119066571A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of general representation learning technology, and in particular to a multi-scale linear general representation method for multivariate time series. Background Art
[0002] In the era of the Internet of Things (IoT), complex dynamic systems such as manufacturing machinery and medical equipment are usually equipped with multiple smart sensors to meet the needs of automation and intelligence. These sensors collect the operating status of the same system from different angles at fixed time intervals to generate multivariate time series. Therefore, multivariate time series in various industries often contain a lot of valuable information, and mining this valuable information is of great significance, so various analysis tasks such as time series classification, prediction, and anomaly detection have received great attention. However, due to the high nonlinearity of dynamic systems, the features hidden in multivariate time series are usually very complex and difficult to capture accurately. In addition, multivariate time series data are usually unlabeled. Therefore, time series analysis tasks often perform poorly.
[0003] Representation learning is a novel feature engineering that can effectively solve the above problems. Specifically, representation learning aims to accurately extract potential features only from unlabeled raw data without being constrained by specific analysis tasks. Therefore, representation learning can improve the performance of downstream analysis tasks in two ways. On the one hand, compared with the original data, the learned representation vector has better discrimination ability. Using the representation vector instead of the original data as input, the analysis task will inevitably have better performance. On the other hand, the more easily recognizable representation vector will further simplify the model size and construction difficulty of the downstream analysis task, and further optimize the speed and performance of the analysis task model. Therefore, it has been proven to be useful to use the feature vector obtained by representation learning instead of the original data as the input of the downstream analysis task. So far, representation learning has been widely explored in the fields of complex networks and computer vision, but it has just started in the field of time series.
[0004] Due to the complexity of multivariate time series, multivariate time series representation learning models based on machine learning and deep learning are currently popular. However, with the rapid development of artificial intelligence and the Internet of Things, multivariate time series in the real world present some new characteristics, which also brings some new challenges to the existing representation models.
[0005] Challenge 1: Complex and strongly coupled features and huge data size. The internal features of multivariate time series are usually hidden in the two dimensions of time and space. In the time dimension, multivariate time series usually show multiple periodic changes, such as daily cycles, weekly cycles and annual cycles, such as Figure 1(a). In the spatial dimension, the data collected by multiple sensors (also called variables) show strong correlation, such as Figure 1 (b) As the demand for intelligence continues to increase, the underlying dynamic complex systems need to perform more and more complex operating modes, which forces them to be equipped with more sensors and perform more intensive data collection. This leads to new characteristics of multivariate time series: more complex features, more variables, stronger variable coupling, larger data scale and scarcer labels. However, traditional representation models are difficult to adapt to these new characteristics, resulting in their inability to accurately capture the complex characteristics of multivariate time series. Therefore, there is an urgent need for a model that can accurately and comprehensively capture the complex characteristics of multivariate time series.
[0006] Challenge 2: Universal representation learning. In the era of the Internet of Things, the types of time series analysis tasks are increasing. Moreover, it is becoming more and more common for multiple tasks to appear in the same scenario at the same time. This forces the multivariate time series representation learning model to support multiple downstream analysis tasks at the same time, rather than only perform well on a single analysis task. However, traditional representation models often fail to provide comprehensive and universal representation features that are suitable for all downstream tasks. Through analysis, it is found that the main reason for this dilemma is that the features learned by these representation models are not comprehensive enough. Specifically, some traditional models only focus on time dependency and ignore periodic changes. Other models only focus on periodic changes at a single scale and ignore periodic changes at multiple scales. Still other models consider periodic changes at multiple scales but ignore spatial correlations between variables. This incomplete representation vector will result in better performance in some analysis tasks and poorer performance in others. Therefore, there is an urgent need to develop a model that can learn more comprehensive and universal feature representations from multivariate time series to support all downstream analysis tasks.
[0007] To overcome the first challenge, existing solutions usually employ deep neural networks with a large number of parameters and deep structures to enhance feature capture capabilities. For example, models based on convolutional neural networks (CNNs) use multi-channel local convolutions and stack multiple convolutional layers to improve feature capture capabilities. Models based on recurrent neural networks (RNNs) stack multiple recursive memory units to improve their effectiveness. Transformer-based models use fine-grained multi-head attention mechanisms to enhance feature extraction capabilities. Although these models are powerful and have strong feature capture capabilities, they often require a lot of computational costs and long training and testing times.
[0008] To address the second challenge, existing solutions attempt to learn more diverse and comprehensive intrinsic features from multivariate time series. For example, the TimesNet model proposed by Zheng et al. is used to learn multi-scale periodic changes, while Choi et al. explored multi-task learning models and proposed the Smate model, which can learn temporal and spatial features simultaneously. The TS2Vec model further learns multi-scale spatiotemporal features from multivariate time series. However, as the complexity of features in multivariate time series increases, these models will become increasingly large and cumbersome, requiring more and more computing and storage resources. Summary of the invention
[0009] In order to solve the above-mentioned technical problems, the present invention provides a multi-scale linear universal representation method for multivariate time series, which uses the lightweight architecture of ALL-MLP to achieve high-speed, high-precision and high-versatility representation learning.
[0010] In order to solve the above technical problems, the technical solution proposed by the present invention is:
[0011] A multi-scale linear universal representation method for multivariate time series is proposed. A representation learning model based on a full multi-layer perceptron architecture is constructed from three aspects: feature extraction, single-scale representation learning and multi-scale representation aggregation. Specifically, firstly, a multi-scale selection strategy is designed to select K representative periods for a multivariate time series according to the frequency domain features. Then, according to the Fourier transform theory, an interpretable Fourier feature extractor is designed to learn temporal features. Next, a "1+M" structured full multi-layer perceptron multi-scale representation network is constructed in conjunction with the Fourier feature extractor to capture the spatiotemporal features of a single scale. Finally, a linear aggregation network is designed to fuse the spatiotemporal features at K scales to generate the final representation of the multivariate time series.
[0012] As a further improvement of the above technical solution:
[0013] Preferably, the universal linear characterization method comprises the following contents:
[0014] Step S1, multi-scale selection strategy
[0015] Use Fast Fourier Transform (FFT) to map the multivariate time series from the time domain to the frequency domain; select K distinguishable amplitudes and convert them to scale values represents K periods in the time domain;
[0016] Step S2: Single-scale linear universal representation learning
[0017] Design a "1+M" structured all-multilayer perceptron linear universal representation learning network, using "1" neural network-based Fourier feature extractor as a shared encoder and "M" two-layer MLPs as private decoders of M variables to form a "1+M" structured autoencoder network for learning single-scale spatiotemporal universal features;
[0018] For a period scale s k , multivariate time series Split into non-overlapping samples Then, each sample is processed by the Fourier feature extractor to obtain the feature The obtained features are input into the dedicated MLP of each variable to reconstruct x k ; When all samples are learned, the features at the current scale are obtained
[0019] After learning K scale representations, K feature sets will be obtained
[0020] Step S3, multi-scale representation aggregation
[0021] Design another "1+M" structure full multi-layer perceptron aggregation network for K features of K scales Fusion to generate an initial general representation Then, the aggregated features are generated through the context aggregation mechanism Next, Z is input into the MLP corresponding to each variable to predict future values, making each timestamp feature smooth and matching the evolution trend of the original time series.
[0022] Preferably, in step S1, the multi-scale selection strategy specifically includes:
[0023] The first step is to split the multivariate time series into N segments, each containing S timestamps: If the length of the multivariate time series is less than S, it is extended to S using nearest neighbor interpolation;
[0024] In the second step, each segment of length S is converted into S / 2 frequency components in the frequency domain using the Fast Fourier Transform FFT:
[0025] The third step is to calculate the average amplitude of each frequency in the M×N segments:
[0026] The fourth step is to sort the average amplitudes in descending order and divide them into K regions. The maximum amplitude is selected from each region to obtain K representative amplitudes:
[0027] In the fifth step, according to formula (1), the K amplitudes are converted into scale values of K representative periods:
[0028]
[0029] Among them, Amp k is the maximum amplitude of the kth region.
[0030] Preferably, in step S2, the design process of the Fourier feature extractor based on the neural network is:
[0031] First, according to the Fourier transform principle, the ideal feature of a multivariate time series is regarded as d sinusoidal basis signals, and each feature signal is represented as follows:
[0032]
[0033] Among them, A j 、w j and b j Represent the amplitude, frequency and phase of the jth characteristic signal respectively;
[0034] Secondly, each feature signal is discretized into T timestamps, which is consistent with the original multivariate time series;
[0035]
[0036] Among them, t ji is the t-th timestamp;
[0037] Then, a one-layer MLP network is constructed with training parameters as and The unique information used to learn each feature signal is as follows:
[0038]
[0039] The first half of the course is about learning cyclical changes, while the second half is about learning trend changes.
[0040] Then, according to Fourier transform theory, the d features of each timestamp are summed to generate a reconstructed value, and the trainable parameters are optimized based on the difference between the reconstructed value and the true value;
[0041] Finally, a feature matrix can be generated for the multivariate time series based on the trained parameters
[0042] Preferably, in step S2, the single-scale representation learning process of the full multi-layer perceptron based on the "1+M" structure is:
[0043] S2-2-1, sampling: according to the current scale length s k , a multivariate time series is divided into multiple lengths s k Non-overlapping fragments of Where N k is the number of fragments; is a sample, where is the value of the mth variable at the i-th timestamp;
[0044] S2-2-2, Encoder: Construct a Fourier feature extractor with d features as s k A shared encoder of M variables at scale; the parameters of the encoder Randomly initialize the parameters From the original multivariate time series Through two linear transformations and To initialize;
[0045] When the nth sample Input to the encoder will generate s in sequence k The d-dimensional features of timestamps When all N k When samples are input into the encoder, a multivariate time series with scale s is obtained. k Spatiotemporal characteristics
[0046] S2-2-3, decoder: M two-layer MLPs are employed as private decoders of M variables, and cooperate with the encoder to form an autoencoder network with a “1+M” structure;
[0047] When the feature of the nth sample When input to the mth MLP, s is generated k reconstructed value and captures the variable m from s k ×(n-1)+1 to s k ×n time dependence; when the shared features of the nth sample When M variables can be accurately reconstructed, from s k ×(n-1)+1 to s k The spatial correlation between M variables at ×n timestamps is captured. When all samples can be accurately reconstructed, at scale s k The common spatiotemporal features under are captured;
[0048] S2-2-4, Loss: Mean square error (MSE) function as the loss function of the "1+M" linear representation network:
[0049]
[0050] in represents the reconstructed value of the mth variable at the ith timestamp, Indicates the true value; N k represents the number of training samples, s k ×n+i represents the time coordinate of the i-th timestamp in the n-th sample.
[0051] Preferably, in step S3, the multi-scale characterization aggregation process is as follows:
[0052] S3-1, Sampling: A multivariate time series of length T is split into T overlapping samples of length P;
[0053] S3-2, Initialization Aggregation: First, the K-scale features obtained from the single-scale universal representation learning Will directly splice along the feature dimension to get Then, a random aggregation matrix is initialized And with Z All Multiply to reduce the dimension and get the initial aggregate features as follows:
[0054] Z INT =W All ×Z All (9)
[0055] S3-3, context aggregation: Perform context aggregation on each timestamp in turn to generate the final aggregated features First, construct and randomly initialize the shared weight matrix Where L represents the context window length; subsequently, a context window is defined for each timestamp And from Z All Extract features for each window Then, the feature Z of each window C With shared weight W C Perform Hadamard product and sum to get new features for each timestamp Finally, the new features of each timestamp are updated to obtain the final aggregated features;
[0056] S3-4, Prediction Learning: Construct another ALL-MLP lightweight prediction network with a "1+M" structure, using aggregated features As shared features, M two-layer MLPs are used as private predictors for M variables; each MLP consists of d input neurons, d hidden neurons, and P output neurons; the aggregated features of each sample will pass through M MLPs to learn the spatial correlation of multiple variables and ensure the trend consistency with the original time series; specifically, the t-th sample will generate P predicted values of [t, (t+P-1)] after the prediction learning of the m-th MLP;
[0057] S3-5, loss: MSE function is used as the loss function:
[0058]
[0059] in, are the predicted value and true value of the jth timestamp in the tth sample, respectively;
[0060] After multi-scale representation aggregation, a multivariate time series The final representation of is generated as
[0061] Compared with the prior art, the multi-scale linear universal characterization method for multivariate time series provided by the present invention has the following advantages:
[0062] (1) The multi-scale linear universal characterization method for multivariate time series of the present invention designs a simple and interpretable feature extractor for extracting time features quickly and accurately. Inspired by the Fourier transform, it represents the multivariate time series as a combination of multiple sine waves and cosine waves. And a linear MLP network is used to simulate the Fourier transform process and automatically learn the spectral information of the wave instead of traditional mathematical calculations. To the best of our knowledge, this is the first work that uses a neural network to simulate the Fourier transform process and use it as a feature extractor.
[0063] (2) The present invention's multi-scale linear universal representation method for multivariate time series develops an unsupervised lightweight representation learning model TS-Linear based on a full multi-layer perceptron architecture, which is used to generate feature representations of multivariate time series more quickly and comprehensively. The TS-Linear model includes a novel multi-scale selection strategy, a single-scale representation learning network based on ALL-MLP, and another representation aggregation network based on ALL-MLP, and adopts a "1+M" learning structure to accurately and efficiently learn multi-scale spatiotemporal features in multivariate time series. To the best of our knowledge, this is the first work that uses a linear model based on ALL-MLP for multivariate time series representation learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] Figure 1(a) shows multiple periodic changes between timestamps 15550 and 15850 on the ETTh1 dataset of the present invention.
[0065] Figure 1 (b) is the correlation between the six variables between timestamps 0 and 100 on the ETTh1 dataset of the present invention.
[0066] Figure 2 This is the framework diagram of the TS-Linear model in the present invention.
[0067] Figure 3 It is a flow chart of the multi-scale selection strategy in the present invention.
[0068] Figure 4 is a schematic diagram of an interpretable Fourier feature extractor in the present invention.
[0069] Figure 5 Schematic diagram of multi-scale characterization aggregation in the present invention.
[0070] Figure 6 It is a performance graph of the multivariate time series prediction task in the verification experiment of the present invention.
[0071] Figure 7 It is a visualization diagram of the 96-step prediction task in the verification experiment of the present invention. DETAILED DESCRIPTION
[0072] The specific embodiments of the present invention are described in detail below. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.
[0073] Multivariate time series is used to model T observations from M sensors on a dynamic continuous system and is defined as From the perspective of time, where x t represents the observation value of the Mth sensor at the tth time stamp. From the perspective of variable (space) dimension, where x m Represents the observation value of the mth sensor at the Tth timestamp.
[0074] The goal of unsupervised multivariate time series representation learning is to learn a function, The original multivariate time series is mapped into a new feature space. In this new space, the mapping vector It's the original data After representation learning, Z can replace X as the input of downstream analysis tasks, thereby simplifying the complexity of the downstream analysis task model and improving the performance of downstream analysis tasks.
[0075] like Figures 1 to 5As shown, the multi-scale linear universal representation method for multivariate time series of the present invention proposes a lightweight universal representation learning model of unsupervised ALL-MLP (full multilayer perceptron) suitable for multivariate time series, referred to as TS-Linear. In order to achieve high speed, TS-Linear abandons the traditional deep structure and huge parameters, and builds a lightweight representation model of ALL-MLP with one or two layers of MLP as the backbone. In order to achieve high precision and comprehensiveness, a Fourier feature extractor based on a neural network and a "1+M" learning structure are designed, and together with the representation model based on ALL-MLP, they are used to learn the spatiotemporal characteristics of multiple period scales in multivariate time series.
[0076] The multi-scale linear universal characterization method for multivariate time series of the present invention includes the following contents:
[0077] Step S1, multi-scale selection
[0078] The multivariate time series is mapped from the time domain to the frequency domain using the Fast Fourier Transform (FFT). Then, K distinguishable amplitudes are selected and converted to scale values represents K periods in the time domain.
[0079] Multivariate time series usually exhibit multiple periodic changes, such as Figure 1 (a). For example, traffic data shows daily and weekly changes, weather data shows daily and seasonal changes, and electricity consumption data shows monthly and annual changes. These multi-periodic changes make the intrinsic characteristics of multivariate time series complex and changeable, making them difficult to accurately extract. Therefore, the features extracted by existing characterization models are not comprehensive or universal enough to be used in all downstream analysis tasks. To solve this problem, this embodiment aims to comprehensively learn the periodic changes of multivariate time series. To achieve this goal, two problems must be solved:
[0080] (1) How to identify periodic changes: According to Fourier transform theory, multivariate time series can usually be interpreted as a combination of sinusoidal signals with obvious periodic patterns. According to this theory, this embodiment applies fast Fourier transform (FFT) to convert multivariate time series from time domain to frequency domain, and selects K representative signals to represent periodic changes.
[0081] (2) How to efficiently and accurately learn multiple periodic changes: Once K representative periods are determined, the goal of this embodiment is to perform independent feature learning for each period.
[0082] This embodiment proposes a multi-scale selection strategy for automatically identifying K representative periodic changes from a multivariate time series, such as Figure 3As shown. In the first step, for the convenience of calculation, the multivariate time series is divided into N segments, each of which contains S timestamps: If the length of the multivariate time series is less than S, it is extended to S using the nearest neighbor interpolation method. In the second step, each segment of length S is converted into S / 2 frequency components in the frequency domain using the fast Fourier transform FFT: The third step is to calculate the average amplitude of each frequency in all generated M×N segments: The fourth step is to sort these average amplitudes in descending order and classify them into K regions. The maximum amplitude is selected from each region to obtain K representative amplitudes: Finally, according to formula (1), the K amplitudes are converted into K scale values representing periodic changes:
[0083]
[0084] Among them, Amp k is the maximum amplitude of the kth region.
[0085] Step S2: Single-scale linear universal representation learning process
[0086] In this embodiment, an ALL-MLP single-scale representation learning network based on a "1+M" structure is designed, using "1" neural network-based Fourier feature extractor as a shared encoder and "M" two-layer MLPs as private decoders of M variables to form a "1+M" autoencoder network. This network can quickly and accurately perform scale-independent single-scale spatiotemporal feature learning. For a period scale s k , multivariate time series First, it is split into non-overlapping samples Then, each sample is processed by the Fourier feature extractor to obtain the feature Then input it into the dedicated MLP for each variable to reconstruct x k The above process can learn the temporal characteristics of each variable, as well as the M variables at scale s. k The variable (spatial) correlation under the condition of After single-scale representation learning at K scales, K spatiotemporal feature sets are obtained.
[0087] The specific steps include:
[0088] S2-1, Fourier feature extractor based on neural network
[0089] In order to match the linear universal representation model based on ALL-MLP, this embodiment designs a simple and highly interpretable feature extractor to enhance the feature capture capability under a lightweight architecture. Specifically, the desired feature extractor needs to have the following two characteristics:
[0090] (1) Feature extractors should be highly interpretable
[0091] Time series are similar to signals, and mature signal analysis methods such as Fourier transform and wavelet transform have strong interpretability. Therefore, this embodiment uses Fourier transform, the simplest and most widely used method, as the basic principle of feature extractor. The essence of Fourier transform theory is to represent nonlinear signals as a combination of infinite sinusoidal basis signals, such as Figure 4 (a). In this case, each basis signal represents a feature of the multivariate time series.
[0092] (2) Feature extractors should be flexible and scalable
[0093] Although the Fast Fourier Transform (FFT) follows a strict mathematical process, the direct use of FFT lacks flexibility and scalability. For example, the Fast Fourier Transform presets the number of features and cannot tolerate noise well. In addition, although FFT is good at capturing periodic changes, it has defects in adapting to the trend changes of signals. To this end, this embodiment proposes a single-layer MLP network to simulate the Fourier transform process. This method improves the flexibility and scalability of feature extraction. In addition, this embodiment integrates the bias matrix into the MLP network for simultaneously learning the trend changes of signals.
[0094] This embodiment designs an interpretable Fourier feature extractor based on a neural network, such as Figure 4 (b), specifically:
[0095] First, according to the Fourier transform principle, the ideal characteristics of the multivariate time series are regarded as d sinusoidal basis signals. The representation of each characteristic signal is as follows:
[0096]
[0097] Among them, A j 、w j and b j Represent the amplitude, frequency and phase of the j-th characteristic signal respectively.
[0098] Then, each feature signal is discretized into T timestamps, which is consistent with the original multivariate time series.
[0099]
[0100] Among them, t ji is the t-th timestamp.
[0101] Next, we construct a single-layer MLP network with trainable parameters and Automatically learn the unique information of each feature signal, as follows:
[0102]
[0103] The first half learns periodic changes, and the second half learns trend changes. According to Fourier transform theory, this process includes summing up the d features of each timestamp to obtain a reconstructed value, and optimizing the trainable parameters in the neural network using stochastic gradient descent SGD or Adma back propagation based on the difference between the reconstructed value and the true value.
[0104] Finally, the feature matrix of the multivariate time series can be generated based on the trained parameters
[0105] The feature extractor of this embodiment has the following main advantages: simplicity, interpretability, flexibility, and scalability, and can simultaneously capture the periodicity and trend characteristics of multivariate time series.
[0106] S2-2, Single-scale linear universal representation learning
[0107] A linear universal representation network based on ALL-MLP is constructed to quickly and accurately learn the spatiotemporal features under a single periodic change. Specifically, the spatiotemporal features include the temporal dependency of adjacent timestamps within a variable and the spatial correlation between multiple variables. To this end, a Fourier feature extractor is used as "1" shared encoder, and "M" two-layer MLPs are used as exclusive decoders for M variables to create a "1+M" structured automatic encoding network, such as Figure 2 (b) as shown.
[0108] To further improve efficiency, this embodiment further optimizes the sampling process. In traditional sampling, each timestamp and its fixed-length lookback window constitute a sample. In this case, a multivariate time series with T timestamps will generate T samples, and there will be a large number of overlapping timestamps between two samples. To solve this problem, this embodiment uses a period scale s k Based on this, the multivariate time series is divided into multiple lengths of s k This method can not only effectively capture periodic changes, but also reduce the number of samples, thereby improving model efficiency.
[0109] The scale s of the k-th period change k Take the example to illustrate, the spatiotemporal feature learning process is as follows:
[0110] S2-2-1, Sampling: A Multivariate Time Series is divided into multiple lengths of s k Non-overlapping fragments of Where N k is the number of fragments. Each fragment is a sample, in which is the value of the mth variable at the ith timestamp.
[0111] S2-2-2, Encoder: This embodiment employs a Fourier feature extractor with d features as the scale s k The shared encoder of the next M variables. Among them, the parameters Random initialization, parameters From the original multivariate time series Through two linear transformations and to initialize.
[0112] When the nth sample Input into the encoder, the d-dimensional features of the i-th timestamp will be generated.
[0113]
[0114] in A f ,w f ,b f are the fth values of parameters A, w and b respectively; t fj and BAIS fj are the values of the parameter t and the fth row and jth column of BAIS respectively.
[0115] When k After the feature generation of timestamps is completed, the complete feature of the nth sample is obtained, that is, Similarly, in all N k After samples are input into the encoder, we will get the scale s k The spatiotemporal characteristics of
[0116] S2-2-3, Decoder: M two-layer MLPs are employed as private decoders of M variables, forming a "1+M" structured autoencoder network with the shared encoder. Specifically, each MLP consists of d input neurons, d hidden neurons with ReLU activation function, and one output neuron. At the same time, the dropout function is used to prevent the MLP from overfitting.
[0117] When the overall characteristics of the nth sample When input to the mth MLP, each timestamp in the sample is reconstructed in turn. For example, the reconstruction process of the i-th timestamp is as follows.
[0118] First, the feature of the i-th timestamp The first layer of the input MLP is transformed as follows:
[0119]
[0120] in and Represents the output and training parameters of the first layer.
[0121] Then, It is then fed into the second layer of the MLP for linear weighted summation, as shown below:
[0122]
[0123] in is the final output of MLP, are the training parameters of the second layer.
[0124] Through the reconstruction learning of the mth MLP, the nth sample will produce s k reconstructed values. When the nth sample is s on the variable m k When the timestamps can be accurately reconstructed, the features Already captured on variable m from s k ×(n-1)+1 to s k ×n. Similarly, when the shared features When all M variables can be accurately reconstructed, we capture the spatial correlation and scale s among the M variables. k The following time period changes.
[0125] S2-2-4, Loss: Mean squared error (MSE) function is used as the loss function of the “1+M” linear universal representation network.
[0126]
[0127] in represents the reconstructed value of the mth variable at the ith timestamp, Represents the true value. N k represents the number of training samples, s k ×n+i represents the time coordinate of the i-th timestamp in the n-th sample.
[0128] It is worth noting that the spatiotemporal feature learning process of K scales is independent and can be executed in parallel. After learning K period scales, we will get a set of K scale spatiotemporal features.
[0129] Step S3, multi-scale representation aggregation
[0130] Design another "1+M" structure ALL-MLP aggregation network to aggregate K features of K scales Fusion to generate an initial general representation Then, the aggregated features are generated through the context aggregation mechanism Next, Z is input into the MLP corresponding to each variable to predict future values, ensuring the smoothness of each timestamp feature and matching the evolution trend of the original time series.
[0131] In this embodiment, an ALL-MLP aggregation network based on a "1+M" structure is used to fuse spatiotemporal features from K scales and obtain the final representation. When multiple sub-features are combined, the resulting aggregated features may appear rough or lack smoothness, which may adversely affect the accuracy of downstream analysis tasks. To address this problem, a contextual aggregation mechanism is designed in this embodiment to update the features of the current timestamp using the features of multiple previous timestamps, thereby ensuring the local smoothness of the final representation. In addition, predictive learning is also used in the "1+M" structure to enhance the smoothness of each timestamp feature so that it is consistent with the evolution trend of the multivariate time series.
[0132] like Figure 5 and Figure 2 As shown in (c), the detailed multi-scale representation aggregation process is as follows:
[0133] S3-1, Sampling: A multivariate time series of length T is split into T overlapping samples of length P. Specifically, each sample consists of a timestamp and the following P-1 timestamps, forming the tth sample in Represents the value of the mth variable at the ith timestamp.
[0134] S3-2, initial aggregation: Initially, the learned spatiotemporal features of K scales It is directly concatenated along the feature dimension Then, randomly initialize an aggregation matrix And with Z All Multiply to reduce the dimension and get the initial aggregate features as follows:
[0135] Z INT =W All ×Z All (9)
[0136] S3-3, context aggregation: Context aggregation is performed sequentially at each timestamp. For the t-th timestamp, the shared weight matrix is first constructed and randomly initialized Where L represents the context window length. Then, define a context window [t-L+1,t], starting from Z All Extract features from These characteristics Z C With W C Perform Hadamard product and sum to get the new feature of the t-th timestamp as follows:
[0137]
[0138] in, W C The i-th column of Represents Z INT Next, the initial aggregate feature z of the t-th timestamp is t Will be new context features Finally, all timestamps are replaced by context features to obtain the final aggregate feature
[0139] S3-4, prediction learning: In order to ensure the smoothness of Z, this embodiment constructs a "1+M" structured ALL-MLP lightweight prediction network. The network uses Z as a shared feature and uses M two-layer MLPs as private predictors for M variables. Each MLP consists of d input neurons, d hidden neurons, and P output neurons. Similarly, the ReLU activation function and dropout mechanism are applied in the hidden layer. When the tth sample is input into the mth MLP, the prediction process of variable m is as follows:
[0140] First, when the tth sample of length P When input into the MLP, the d-dimensional features of its first timestamp It is input into the first layer of MLP and converted as follows:
[0141]
[0142] in is the output, are the trainable parameters of the first layer.
[0143] Then, It is input into the second layer and linearly weighted summed as follows:
[0144]
[0145] in is the final output of MLP, are the trainable parameters of the second layer.
[0146] After the prediction learning of the mth MLP, the tth sample will generate P prediction values, corresponding to the timestamp [t, (t+P-1)]. When all samples can accurately predict variable m, the aggregate feature Z is smoothed on variable m. When M variables are accurately predicted, the aggregate feature Z has captured the spatial correlation between the M variables and the temporal evolution trend within a single variable.
[0147] S3-5, loss: MSE function is used as the loss function:
[0148]
[0149] in, are the predicted value and true value of the j-th timestamp in the t-th sample respectively;
[0150] After multi-scale representation aggregation, the multivariate time series The final representation is generated as Experimental verification
[0151] 1. Tasks, Datasets, and Evaluation Metrics
[0152] This example selects three popular analysis tasks and 16 publicly available datasets for experiments, each of which has specific evaluation indicators to evaluate performance. The details are summarized in Table 1.
[0153] Table 1 Tasks, datasets and evaluation metrics
[0154]
[0155] 1) Forecasting task: Time series forecasting aims to discover patterns and trends in existing time series data, so as to make accurate predictions for the future. Specifically, historical data is used to build a model, which is then used to predict future values. This example selects MSE (Mean Squared Error) and MAE (Mean Absolute Error) as evaluation indicators. Generally speaking, the lower the MSE and MAE values, the better the model performance.
[0156] To accomplish the prediction task, six public multivariate time series datasets were selected for this validation experiment. The Air quality dataset contains air quality data collected at the road level in a heavily polluted Italian city. The ETT (Electric Transformer Temperature) dataset includes “Oil Temperature” and six electrical load attributes, and three subsets (ETTh1, ETTh2, and ETTm1) were selected. The SML dataset consists of monitoring data from smart homes. The Power dataset is a dataset of electricity consumption in the city of Dettun, which is collected through seven sensor nodes and one edge node in a star configuration.
[0157] 2) Classification task: Time series classification is widely used in various fields. This task aims to train a classifier using labeled data to identify the category of a given time series. In order to evaluate the performance of the model, ACC and AUPRC are selected as the measurement indicators. The higher the value of ACC or AUPRC, the better the performance.
[0158] In the classification task, five widely used subsets (Chinatown, ItalyPowerDemand, RacketSports, ArrowHead, and Libras) are selected from the UCR Time Series Classification Archive as datasets.
[0159] 3) Anomaly Detection Task: Time series anomaly detection aims to identify abnormal events or behaviors in time series datasets. Its focus is on detecting data points that deviate significantly from normal patterns and trends. In order to comprehensively evaluate the performance, F1, ACC, Precision, Recall, and AUC are used as metrics. The higher the values of these metrics, the better the performance.
[0160] In the anomaly detection task, five different real-world datasets were selected. The SMAP (Soil Moisture Active Passive Satellite) dataset includes soil samples and telemetry data. The MSL (Mars Science Laboratory Rover) dataset focuses on the sensor and actuator data of the Mars rover. The SMD (Server Machine Dataset) includes data from 28 different machines, each with 38 dimensions. The MBA (MIT-BIH Supraventricular Arrhythmia Database) dataset contains ECG records of five patients. The ADFP (Anomaly Detection Fallers) dataset originates from an independent living smart home environment.
[0161] 2. Comparison Models
[0162] In order to evaluate the performance of TS-Linear, eight well-known models were selected as comparison models, as shown in Table 2. Among them, TS2Vec and Triplet_Loss models are based on convolutional neural networks (CNN), TNC and CPC models are based on recurrent neural networks (RNN), and CRT models combine CNN and RNN networks. In addition, TST, PatchTST, and TS-TCC are based on Transformer networks.
[0163] For the prediction task, this example selects the TS2Vec, TNC, TST, PatchTST, CPC, TS-TCC, and Triplet_Loss models. In the classification task, all models in the prediction task are used except the TS-TCC model. For anomaly detection, CRT and all models used in the prediction task are used.
[0164] Table 2 Eight comparison models
[0165]
[0166] 3. Performance Analysis
[0167] This embodiment selects three analysis tasks as the basis to compare the performance of TS-Linear and other comparison models. Specifically, the feature representations generated by all models replace the original multivariate time series, and use logic regression, SVM and L1 distance to perform multivariate time series prediction, classification and anomaly detection tasks respectively. It is worth noting that in order to further evaluate the performance changes in short-term and long-term prediction scenarios, this embodiment performs five prediction tasks with different step lengths (24 steps, 48 steps, 96 / 168 steps, 288 / 336 steps, 672 / 720 steps).
[0168] 1) Multivariate time series forecasting tasks: Figure 6 The MSE and MAE performance of eight models on six datasets are shown. Some important conclusions can be drawn from the figure.
[0169] Considering the overall performance: TS-Linear consistently outperforms other comparison models in most cases. TS2Vec performs worse on some datasets. For example, on the ETTh1 dataset, TS-Linear has an average MSE of 0.6151 and an average MAE of 0.5728 over five prediction lengths, while TS2Vec has an average MSE of 0.7468 and an average MAE of 0.6445. In addition, TS-Linear's MSE and MAE on the six datasets are also more stable.
[0170] Considering the prediction step length: As the prediction length increases, the MSE and MAE of all models increase, and the prediction accuracy decreases. TS-Linear and TS2Vec show higher stability compared to other models. For example, on the Air dataset, TS-Linear performs best in 24-step and 672-step predictions, while TS2Vec ranks second in 24-step predictions but last in 672-step predictions.
[0171] Focus on dataset properties: MSE and MAE of all models on different datasets vary significantly. For example, the average MAE of TS-Linear on the ETTh2 dataset is 0.7978, while it drops to 0.6607 on the Power dataset. In addition, datasets with more variables generally perform worse. For example, on the Air dataset, the average MSE of all models is 1.2644, while it is 0.6151 on the ETTh1 dataset.
[0172] In order to further evaluate the prediction accuracy, this example generates visualization graphs of six models on the ETTm1 and SML datasets (see Figure 7 ). In these figures, the red line represents the prediction result and the black line represents the true value. It can be seen from the figure that TS-Linear is more consistent with the true value than other models. Specifically, in the SML dataset visualization, TS-Linear is very consistent with the true value on the left and right sides of the red curve, which shows that TS-Linear is more accurate than other models.
[0173] Table 3 Multivariate time series classification task performance
[0174]
[0175] 2) Multivariate time series classification task: Table 3 shows the classification performance (ACC and AUPRC) of seven models on five datasets. The bold text in the table indicates the best performance of each indicator on a specific dataset, and the underlined text indicates the second best performance. From the table, we can make the following observations.
[0176] Considering the overall performance: TS-Linear always ranks first, followed by TS2Vec, while TST usually performs the worst. For example, on the Chinatown dataset, TS-Linear's ACC score is 0.9971, TS2Vec is 0.9796, and TST is 0.9125. Across all datasets, the average ACC score of TS-Linear is 0.9144, TS2Vec is 0.8949, and TST is 0.8019. In addition, compared with other models, TS-Linear's ACC and AUPRC scores on different datasets are stable.
[0177] Considering different datasets: The performance of all seven models on different datasets varies greatly. For example, TS-Linear has an AUPRC of 0.8907 on the ArrowHead dataset, while it drops to 0.8234 on the Libras dataset. In addition, the performance of all models tends to decrease as the number of variables and categories in multivariate time series datasets increases. For example, on the Air dataset with 24 variables and 2 categories, TS-Linear has an AUPRC of 0.9998. However, on the Libras dataset with 90 variables and 15 categories, the AUPRC drops to 0.8234, the lowest among the five datasets.
[0178] Table 4 Multivariate time series anomaly detection performance
[0179]
[0180]
[0181] 3) Multivariate time series anomaly detection task: Table 4 lists the performance of the nine models on the anomaly detection task on five datasets. The bold text in the table indicates the best performance of each indicator on a given dataset, and the underlined text indicates the second best performance. The following results can be found in Table 5.
[0182] Considering the overall performance: TS-Linear consistently has a significant advantage in most cases. For example, on the MSL dataset, TS-Linear's AUC score is 0.9768, surpassing the second-highest score of 0.9758 of TS-TCC. Even when TSLinear did not get the highest score, it usually ranked second. For example, on the MBA dataset, TS-Linear's accuracy score is 0.8599, slightly lower than TS2Vec's 0.8704. On all datasets, TS-Linear has the best average AUC score of 0.9474.
[0183] Considering stability: TS-Linear shows better stability, always maintaining the first or second ranking in the five datasets and all indicators. For example, on the SMAP dataset, TS-Linear has the highest F1 score of 0.6060, while TST performs the worst with only 0.1389. Similarly, on the MSL dataset, TS-Linear maintains its leading position with an F1 score of 0.7851, while TST has a score of 0.6509. In addition, on the SMAP and MSL datasets, the AUC scores of TS-Linear differ by only 0.0094, while TS2Vec differs by 0.1215.
[0184] In the three experiments, the representation vectors generated by the TS-Linear model always have excellent and stable performance. A large number of experiments also prove that the representation vectors generated by TS-Linear are more comprehensive and can be applied to three different analysis tasks at the same time.
[0185] The present invention presents a multi-scale linear universal representation learning model TS-Linear suitable for multivariate time series. Compared with existing models, TS-Linear has better performance, better interpretability and more comprehensive feature capture capabilities. TS-Linear uses a double-layer MLP as the backbone to build a lightweight representation learning model of ALL-MLP, abandoning the traditional deep neural network with deep structure and huge parameters, thereby improving the speed of the model. According to the Fourier transform theory, TS-Linear designs a highly interpretable Fourier feature extractor to make up for the lack of accuracy caused by abandoning the deep network. In addition, in order to achieve high precision and high versatility, TS-Linear also designs a reconstruction network and a prediction network of ALL-MLP with a "1+M" structure to learn spatiotemporal features of multiple periodic scales and generate the final universal representation. Extensive experiments on three tasks and sixteen public datasets show that TS-Linear outperforms several well-known models.
[0186] The above implementation cases are only preferred embodiments of the present invention and are not intended to limit the present invention in any form. Although the present invention has been disclosed as above with preferred embodiments, they are not intended to limit the present invention. Therefore, any simple modification, equivalent changes and modifications made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.
Claims
1. A multi-scale linear universal representation method for multivariate time series, characterized by: A representation learning model based on the full multi-layer perceptron architecture is constructed from three aspects: feature extraction, single-scale representation learning and multi-scale representation aggregation. Specifically, first, a multi-scale selection strategy is designed to select K representative periods for a multivariate time series based on frequency domain features. Then, based on Fourier transform theory, an interpretable Fourier feature extractor is designed to learn temporal features. Next, a "1+M" structured full multi-layer perceptron multi-scale representation network is constructed in conjunction with the Fourier feature extractor to capture the spatiotemporal features of a single scale. Finally, a linear aggregation network is designed to fuse the spatiotemporal features at K scales to generate the final representation of the multivariate time series.
2. The multi-scale linear universal characterization method for multivariate time series according to claim 1, characterized in that: The linear general characterization method includes the following contents: Step S1, multi-scale selection strategy Use Fast Fourier Transform (FFT) to map the multivariate time series from the time domain to the frequency domain; select K distinguishable amplitudes and convert them to scale values represents K periods in the time domain; Step S2: Single-scale linear universal representation learning Design a "1+M" structured all-multilayer perceptron linear universal representation learning network, using "1" neural network-based Fourier feature extractor as a shared encoder and "M" two-layer MLPs as private decoders of M variables to form a "1+M" structured autoencoder network for learning single-scale spatiotemporal universal features; For a period scale s k , multivariate time series Split into non-overlapping samples Then, each sample is processed by the Fourier feature extractor to obtain the feature The obtained features are input into the dedicated MLP of each variable to reconstruct x k ; When all samples are learned, the features at the current scale are obtained After learning K scale representations, K feature sets will be obtained Step S3, multi-scale representation aggregation Design another "1+M" structure full multi-layer perceptron aggregation network for K features of K scales Fusion to generate an initial general representation Then, the aggregated features are generated through the context aggregation mechanism Next, Z is input into the MLP corresponding to each variable to predict future values, making each timestamp feature smooth and matching the evolution trend of the original time series.
3. The multi-scale linear universal characterization method for multivariate time series according to claim 2, characterized in that: In step S1, the specific process of the multi-scale selection strategy is as follows: The first step is to split the multivariate time series into N segments, each containing S timestamps: If the length of the multivariate time series is less than S, it is extended to S using nearest neighbor interpolation; In the second step, each segment of length S is converted into S / 2 frequency components in the frequency domain using the Fast Fourier Transform FFT: The third step is to calculate the average amplitude of each frequency in the M×N segments: The fourth step is to sort the average amplitudes in descending order and divide them into K regions. The maximum amplitude is selected from each region to obtain K representative amplitudes: In the fifth step, according to formula (1), the K amplitudes are converted into scale values of K representative periods: Among them, Amp k is the maximum amplitude of the kth region.
4. The multi-scale linear universal characterization method for multivariate time series according to claim 3, characterized in that: In step S2, the design process of the Fourier feature extractor based on the neural network is as follows: First, according to the Fourier transform principle, the ideal feature of a multivariate time series is regarded as d sinusoidal basis signals, and each feature signal is represented as follows: Among them, A j 、w j and b j Represent the amplitude, frequency and phase of the jth characteristic signal respectively; Secondly, each feature signal is discretized into T timestamps, which is consistent with the original multivariate time series; Among them, t ji is the t-th timestamp; Then, a one-layer MLP network is constructed with training parameters as and The unique information used to learn each feature signal is as follows: The first half of the course is about learning cyclical changes, while the second half is about learning trend changes. Then, according to Fourier transform theory, the d features of each timestamp are summed to generate a reconstructed value, and the trainable parameters are optimized based on the difference between the reconstructed value and the true value; Finally, a feature matrix can be generated for the multivariate time series based on the trained parameters 5. The multi-scale linear universal characterization method for multivariate time series according to claim 2, characterized in that: In step S2, the single-scale representation learning process of the full multi-layer perceptron based on the "1+M" structure is: S2-2-1, sampling: according to the current scale length s k , a multivariate time series is divided into multiple lengths s k Non-overlapping fragments of Where N k is the number of fragments; is a sample, where is the value of the mth variable at the i-th timestamp; S2-2-2, Encoder: Construct a Fourier feature extractor with d features as s k A shared encoder of M variables at scale; the parameters of the encoder Randomly initialize the parameters From the original multivariate time series Through two linear transformations and To initialize; When the nth sample Input to the encoder will generate s in sequence k The d-dimensional features of timestamps When all N k When samples are input into the encoder, a multivariate time series with scale s is obtained. k Spatiotemporal characteristics S2-2-3, decoder: M two-layer MLPs are employed as private decoders of M variables, and cooperate with the encoder to form an autoencoder network with a "1+M" structure; When the feature of the nth sample When input to the mth MLP, s is generated k reconstructed value and captures the variable m from s k ×(n-1)+1 to s k ×n time dependence; When the shared features of the nth sample When M variables can be accurately reconstructed, from s k ×(n-1)+1 to s k The spatial correlation between M variables at ×n timestamps is captured; when all samples can be accurately reconstructed, at scale s k The common spatiotemporal features under are captured; S2-2-4, Loss: Mean Squared Error (MSE) function as the loss function of the "1+M" linear representation network: in represents the reconstructed value of the mth variable at the ith timestamp, Indicates the true value; N k represents the number of training samples, s k ×n+i represents the time coordinate of the i-th timestamp in the n-th sample.
6. The multi-scale linear universal characterization method for multivariate time series according to claim 2, characterized in that: In step S3, the multi-scale characterization aggregation process is as follows: S3-1, Sampling: A multivariate time series of length T is split into T overlapping samples of length P; S3-2, Initialization Aggregation: First, the K-scale features obtained from the single-scale universal representation learning Will directly splice along the feature dimension to get Then, a random aggregation matrix is initialized And with Z All Multiply to reduce the dimension and get the initial aggregate features as follows: WITH INT =In All ×Z All (9) S3-3, context aggregation: perform context aggregation on each timestamp in turn to generate the final aggregated features First, construct and randomly initialize the shared weight matrix Where L represents the context window length; Then, a context window is defined for each timestamp And from Z All Extract features for each window Then, the feature Z of each window C With shared weight W C Perform Hadamard product and sum to get new features for each timestamp Finally, the new features of each timestamp are updated to obtain the final aggregated features; S3-4, Prediction Learning: Build another "1+M" structure of a fully multi-layer perceptron lightweight prediction network, using aggregated features As shared features, M two-layer MLPs are used as private predictors for M variables; each MLP consists of d input neurons, d hidden neurons, and P output neurons; the aggregated features of each sample are passed through M MLPs to learn the spatial correlation of multiple variables and are consistent with the original time series trend; The t-th sample undergoes prediction learning of the m-th MLP, generating P prediction values of [t, (t+P-1)]; S3-5, loss: MSE function is used as the loss function: in, are the predicted value and true value of the jth timestamp in the tth sample, respectively; After multi-scale representation aggregation, a multivariate time series The final representation of is generated as
Citation Information
Patent Citations
A multivariate time sequence-based feature fusion method
CN113361590A
Multivariable time sequence prediction method based on maximized mutual information entropy improvement
CN116976490A
