Multi-scale time sequence feature extraction method based on local linear layer

Abstract

The invention provides a multi-scale time sequence feature extraction method based on a local linear layer. The method comprises the following steps: S1, carrying out standardization processing on an input univariate or multivariate time sequence; s2, setting a plurality of time windows with different sizes, wherein the size of each window corresponds to one scale; s3, under a single scale, retaining a local connection structure in a time dimension by adopting a mask mechanism, and independently extracting local features of time steps in each window in the scale through linear transformation; s4, under a single scale, sharing a linear weight in a channel dimension, and carrying out special extraction on all variables by adopting the same linear weight matrix; and S5, stacking the initial sequence and the feature sequence extracted under each scale along the dimension of the scale to form a multi-scale feature sequence of each variable, and outputting the multi-scale feature sequence to be used by a subsequent model. The invention provides a novel neural network layer named as a local linear layer, local modeling and multi-scale feature extraction can be realized, parameter efficiency and modeling capability can be improved, and the method is suitable for time sequence modeling tasks.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and time series modeling technology, and specifically relates to a method based on a local linear layer (Local Linear Layer, represented by L 3 ) is a multi-scale time series feature extraction method. Background Art

[0002] Multidimensional time series often exhibit complex temporal features and strong inter-variable correlations. Existing multivariate time series modeling methods have limitations in extracting local temporal features and modeling inter-variable relationships. Traditional standard convolutional layers extract local features in the temporal dimension using sliding windows, often assuming independence between channels. This makes them inadequate for time series data with high inter-variable correlations. Furthermore, the convolution operation introduces multiple independent filters in the channel dimension, which can lead to large parameter counts and high model complexity. While pooling layers can reduce computational complexity, they lose critical fine-grained temporal information, making them inadequate for processing sequence data with abrupt or irregular structures. Existing linear or fully connected layers, while capable of expressing information, lack the ability to model local structure. Therefore, a simple, parameter-efficient neural network architecture is urgently needed to improve time series modeling performance, particularly in high-dimensional scenarios. Summary of the Invention

[0003] The purpose of the present invention is to propose a multi-scale time series feature extraction method based on a local linear layer, wherein the "local linear layer" is an artificial neural network layer proposed by the present invention in which output neurons are linearly associated with local input neurons.

[0004] The technical solution of the present invention is: a multi-scale time series feature extraction method based on a local linear layer, characterized by comprising the following steps:

[0005] S1: Standardization: Standardize the input single variable or multivariate time series, where "variable" is also called "channel";

[0006] S2: Multi-scale window construction: set multiple time windows of different sizes, each window size corresponds to a scale;

[0007] S3: Local feature extraction in the temporal dimension: At a single scale, a masking mechanism is used within the temporal dimension to preserve the local linear connection structure, and local features are independently extracted for each time step in each window within that scale through linear transformation.

[0008] S4: Channel dimension weight sharing: In a single scale, linear weights are shared within the channel dimension, and the same linear weight matrix is ​​used for feature extraction for all variables;

[0009] S5: Multi-scale feature stacking and output: The initial sequence and the feature sequences extracted at each scale are stacked along the scale dimension to form a multi-scale feature sequence for each variable, which is output for use by subsequent models.

[0010] Furthermore, step S1 includes the following sub-steps:

[0011] S11. Time series of initial acquisition Each channel of is normalized individually:

[0012]

[0013] where μ (i) ,σ (i) is the mean and standard deviation of the i-th variable; is the initial sequence and standardized sequence of the i-th variable; is a set of real numbers, L is the sequence length, and C is the number of variables, also called the number of channels;

[0014] S12. Introducing channel-specific trainable affine parameters:

[0015]

[0016] where γ i is the scaling factor, β i are translation factors, which are updated as affine parameters during model training.

[0017] Furthermore, step S2 includes the following sub-steps:

[0018] S21. Set the window size set W = [w1, w2, ..., w G-1 ], where w t is the window size of the tth scale, G is the total number of scales, including the initial sequence. The window size increases sequentially to gradually aggregate information on larger scales;

[0019] S22, each scale t (t ≥ 1) is associated with an independent trainable weight matrix and bias where L′=L+w t -1 represents the length of the sequence after padding.

[0020] Furthermore, step S3 includes the following sub-steps:

[0021] S31, the window size w of the t-th scale t , calculate the filling length For the sequence x of variable i ( i) Perform symmetric filling operations:

[0022]

[0023] in is the padded sequence at the tth scale. This padding method maintains the physical continuity of the time series and ensures that the output length remains equal to the input length when the window interval step size S is 1;

[0024] S32. Define the mask matrix M t ∈{0,1} L′×L , where the mask vector of the kth column is for:

[0025]

[0026] Where l represents the lth time step, k∈[0,N] represents the index of the output neuron, and N is the number of output neurons, that is, the number of windows at a single scale. When S = 1, the output length remains equal to the input length. This mask matrix is ​​used to restrict each output neuron to only connect to the local input window;

[0027] S33, weight matrix V t Apply a mask:

[0028]

[0029] Where ⊙ is the corresponding multiplication of elements at the same position in two matrices of the same shape and size;

[0030] S34, filling sequence Perform a linear change:

[0031]

[0032] in is the feature sequence at the t-th scale.

[0033] Furthermore, step S4 includes the following sub-steps:

[0034] S41. When the input is a univariate sequence, that is, there is only one channel, the parameters of the current scale are directly used for processing:

[0035]

[0036] S42. When the input is a multivariate sequence, all variables i=1,2,…,C share a set of parameters Perform a linear transformation on each variable independently:

[0037]

[0038] LinearTransform(·) refers to the linear transformation in S34. Parameter sharing reduces computational complexity and accommodates inter-variable correlations. The operations of S3 and S4 on a single scale essentially pass the sequence through a local linear layer of a specific window size.

[0039] Furthermore, step S5 includes the following sub-steps:

[0040] S51. Stack the features of the initial sequence scale (t=0) and other scales (t≥1) into a multi-scale representation:

[0041]

[0042] where x (i) is the initial sequence, is the characteristic sequence of the t-th scale;

[0043] S52. Output the multi-scale feature set of all variables:

[0044]

[0045] Each Contains the multi-scale feature sequence of variable i;

[0046] S53, will x w Input to subsequent models to support time series forecasting or other related tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Flow chart of the method of the present invention.

[0048] Figure 2 The forecasting performance of local linear layers using different sets of window sizes is shown on three representative time series datasets.

[0049] Figure 3 The prediction performance of four representative time series datasets using local linear layers, convolutional layers, and average pooling layers is demonstrated.

[0050] Figure 4 The difference between the local linear layer neural network proposed in this invention and the standard 1D convolution is demonstrated.

[0051] Figure 5 The implementation mechanism of the local linear layer proposed in this invention is demonstrated: linear masking is performed on the standard fully connected layer.

[0052] Figure 6The weights of local linear layers of different window sizes are initialized using the inverse of the window size, and examples of weight heat maps and multi-scale feature sequences after training are shown. DETAILED DESCRIPTION

[0053] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0054] like Figure 1 As shown, the present invention provides a multi-scale time series feature extraction method based on a local linear layer, comprising the following steps:

[0055] S1: Standardization: Standardize the input single variable or multivariate time series, where "variable" is also called "channel";

[0056] S2: Multi-scale window construction: set multiple time windows of different sizes, each window size corresponds to a scale;

[0057] S3: Local feature extraction in the temporal dimension: At a single scale, a masking mechanism is used within the temporal dimension to preserve the local linear connection structure, and local features are independently extracted for each time step in each window within that scale through linear transformation.

[0058] S4: Channel dimension weight sharing: In a single scale, linear weights are shared within the channel dimension, and the same linear weight matrix is ​​used for feature extraction for all variables;

[0059] S5: Multi-scale feature stacking and output: The initial sequence and the feature sequences extracted at each scale are stacked along the scale dimension to form a multi-scale feature sequence for each variable, which is output for use by subsequent models.

[0060] In this paper, we used the proposed multi-scale time series feature extraction method based on local linear layers to perform long time series forecasting on nine time series datasets spanning electricity, meteorology, finance, energy, and transportation, with significant differences in data characteristics and number of variables. The datasets are described in Table 1:

[0061] Table 1:

[0062]

[0063] In the implementation of the present invention, the long-term prediction task is unified as input length L = 96, prediction length F = [96, 192, 336, 720], and the prediction accuracy is evaluated by mean square error (MSE) and mean absolute error (MAE):

[0064]

[0065] In the embodiment of the present invention, step S1 includes the following sub-steps:

[0066] S11. Time series of initial acquisition Each channel of is normalized individually:

[0067]

[0068] where μ (i) ,σ (i) is the mean and standard deviation of the i-th variable; is the initial sequence and standardized sequence of the i-th variable; is a set of real numbers, L is the sequence length, and C is the number of variables, also called the number of channels. In this implementation, L = 96. For different datasets, refer to Table 1 for different variable dimensions C.

[0069] S12. Introducing channel-specific trainable affine parameters:

[0070]

[0071] where γ i is the scaling factor, β i is the translation factor. In the implementation, we use the initialization γ i =1,β i = 0, and learn these two affine parameters through back propagation.

[0072] Furthermore, step S2 includes the following sub-steps:

[0073] S21. Set the window size set W = [w1, w2, ..., w G-1 ], where w t is the window size at the tth scale, and G is the total number of scales, including the initial sequence. The window size increases sequentially to aggregate information at progressively larger scales. In this implementation, W = [3, 7, 15, 27, 43] and its subsets are used, with the window size increasing in powers of 2.

[0074] For low-dimensional datasets (such as ETT, Exchange, Weather, with channel dimensions less than 100), W = [3, 7, 15, 27] is usually used. For high-dimensional datasets (such as Solar-Energy, Electricity, Traffic, with channel dimensions greater than or equal to 100), W = [3, 7, 15] or W = [3, 7] is usually used to balance accuracy and computational efficiency.

[0075] S22, each scale t (t ≥ 1) is associated with an independent trainable weight matrix and bias where L′=L+w t -1 represents the length of the sequence after padding.

[0076] Furthermore, step S3 includes the following sub-steps:

[0077] S31, the window size w of the t-th scale t , calculate the filling length For the sequence x of variable i (i) Perform a symmetric fill operation:

[0078]

[0079] in is the padded sequence at the t-th scale. In implementation, for example, t =15, calculate p t =7, perform symmetrical filling:

[0080]

[0081] This filling method maintains the physical continuity of the time series and ensures that the output length remains equal to the input length when the window interval step size S is 1;

[0082] S32. Define the mask matrix M t ∈{0,1} L′×L , where the kth column for:

[0083]

[0084] Where l represents the lth time step, k∈[0,N] represents the index of the output neuron, and N is the number of output neurons, that is, the number of windows. When S=1, the output length remains equal to the input length.

[0085] In implementation, for example, for S=1, w t = 15, output position k = 50 is connected only to inputs l = 50 to 64:

[0086]

[0087] The mask matrix is ​​used to restrict each output neuron to connect only to the local input window. At this time, the trainable parameters of the sparse connection layer are only 15.6% of those of the fully connected layer.

[0088] S33, weight matrix V t Apply a mask:

[0089]

[0090] Where ⊙ is the corresponding multiplication of elements at the same position in two matrices of the same shape and size;

[0091] S34, set S=1 in the implementation, fill the sequence Perform a linear change:

[0092]

[0093] in is the feature sequence at the t-th scale.

[0094] Furthermore, step S4 includes the following sub-steps:

[0095] S41. When the input is a univariate sequence, that is, there is only one channel, the parameters of the current scale are directly used for processing:

[0096]

[0097] S42. When the input is a multivariate sequence, all variables i=1,2,…,C share a set of parameters Perform a linear transformation on each variable independently:

[0098]

[0099] LinearTransform(·) refers to the linear transformation in S34. Parameter sharing reduces computational complexity and accommodates inter-variable correlations. The operations of S3 and S4 on a single scale essentially pass the sequence through a local linear layer of a specific window size.

[0100] For example, in the implementation of the Electricity dataset, its channel C = 321 and the window size w t =15, the number of parameters of the standard 1D convolution is 1.54M, while the number of parameters of this local linear layer is only 10.5K.

[0101] Furthermore, step S5 includes the following sub-steps:

[0102] S51. Stack the features of the initial sequence scale (t=0) and other scales (t≥1) into a multi-scale representation:

[0103]

[0104] where x (i) is the initial sequence, is the characteristic sequence of the t-th scale;

[0105] S52. Output the multi-scale feature set of all variables:

[0106]

[0107] Each Contains the multi-scale feature sequence of variable i;

[0108] S53, will x w Input to subsequent models to support time series forecasting or other related tasks. During the implementation process, the subsequent models used two types of models: Transformer (a neural network model based on the attention mechanism) and linear structure (a neural network model based on fully connected layers).

[0109] To verify the effectiveness of the local linear layer in the present invention, the effects of different window selections on the model prediction accuracy were tested when the subsequent model adopted the Transformer structure. Figure 3 The prediction accuracy of the model is shown as the window changes on three representative data sets (Weather dimension 21, Solar-Energy dimension 137, Electricity dimension 321), and the average of the results of four prediction lengths is taken. It shows that when the local linear layer is cancelled, the prediction accuracy of all prediction models is the worst; when the multi-scale window based on the local linear layer is introduced, the prediction ability of the model first decreases and then fluctuates. The window selection is less sensitive to the Weather data set, and is more sensitive to the Solar-Energy data set and the Electricity data set. Under the best window selection, compared with no multi-scale, the MSE on the Weather data set is reduced from 0.242 to 0.238, on the Solar-Energy data set is reduced from 0.216 to 0.192, and on the ECL (i.e. Electricity) data set is reduced from 0.172 to 0.164. It is verified that the multi-scale feature extraction based on the local linear layer significantly enhances the model performance, and the effectiveness of the present invention is verified.

[0110] In order to compare the efficiency advantage of the local linear layer in the present invention compared with convolution and (average) pooling, the parameter amount, running time and GPU memory usage of a single-layer local linear layer, standard convolution and average pooling are compared in the implementation. Table 2 shows the performance of the three neural network layers for different dimensional data sets and different window settings. In low-dimensional scenarios (ETT channel dimension is 7), the three show similar running time and resource usage. In high-dimensional scenarios (ECL channel dimension is 321), the convolution parameters grow explosively, while the parameter amount of the local linear layer is only related to the sequence length and window size, and has nothing to do with the number of channels, thereby maintaining fewer parameters. At the maximum setting, its trainable parameters are only 0.00093 times that of the convolution. The running time and resource usage of the local linear layer in high-dimensional scenarios are similar to those of the average pooling, which is significantly better than the convolution layer.

[0111] Table 2:

[0112]

[0113] In order to compare the feature extraction capabilities of the local linear layer in the present invention compared with the convolutional layer and the average pooling layer, the prediction performance of a single local linear layer, a standard convolutional layer, and an average pooling layer on four representative data sets (Solar-Energy, ECL, Weather, and ETTm2) was compared in the implementation. Figure 3 The average MSE and MAE results of the three neural network layers at four prediction lengths are shown. The local linear layer performs best on all datasets in terms of both MAE and MSE indicators. In addition, the performance of average pooling and convolution fluctuates for datasets with significantly different distribution characteristics. Average pooling is relatively robust on stable series, but it has difficulty capturing key information when dealing with non-stationary datasets (such as ETTm2); convolution is less effective when dealing with multivariate datasets with strong correlation between variables (such as ECL). In contrast, the local linear layer can adapt to the complex local temporal dynamics and strong correlation between variables in the time series, and thus shows stable performance on all datasets.

[0114] The subsequent models of the above experiments all use the Transformer structure. In order to verify the versatility of the local linear layer in the present invention, it can be integrated into various models as a basic neural network layer. In the implementation, it is integrated into the long-term prediction model based on the linear structure, and a downsampling method with a step size of S=2 is used to gradually extract multi-scale features. Table 3 shows the average prediction performance of the local linear layer, convolution layer and average pooling layer on four representative data sets. The local linear layer continues to show stable performance improvement on each representative data set, while the standard convolution performs poorly on the high-dimensional data set ECL.

[0115] Table 3:

[0116]

[0117] The above experiments demonstrate the effectiveness and efficiency of the present invention, especially the potential of the local linear layer as a universal network structure. In order to theoretically explain the mechanism of action of the local linear layer in the present invention, Figure 4A diagram comparing the learning mechanisms of local linear layers and standard 1D convolutions is shown. Local linear layers use different weight kernels (windows) in the time dimension and share weights across channels (variables), which is the opposite of the convolutional layer's model of sharing weights in the time dimension and learning independently across channels. Local linear layers ensure that each output neuron focuses only on adjacent input neurons through local connections, allowing them to adapt to local temporal dynamics. At the same time, local linear layers share parameters across all channels to model inter-variable correlations. By adjusting the window size, local linear layers can effectively extract information at different time scales. It is worth noting that the number of parameters in local linear layers is independent of the number of variables, resulting in significant parameter efficiency advantages over convolutional layers when processing high-dimensional time series.

[0118] To intuitively illustrate how the local linear layer is implemented, Figure 5 This paper demonstrates the mechanism by which a local linear layer is implemented through masking of a fully connected layer. A local linear layer is a linear layer designed to capture local dynamics, achieving local modeling by retaining only the local connections within a temporal window. As the window size increases, the local linear layer can progressively approximate a fully connected linear embedding layer, thus inheriting the convergence guarantees of the linear projection of the fully connected layer.

[0119] In order to visualize the mechanism of multi-scale feature extraction by local linear layer, Figure 6 We present heatmaps of local linear layers initialized with the inverse of the window size on the ETTh2 dataset, with window sizes of [3, 7, 15, 27]. The heatmaps of the local linear layers reveal a more flexible weight learning pattern than traditional pooling methods that rely on fixed parameters. As the window size increases, the curves from Scale 0 to Scale 5 clearly evolve from fine-grained fluctuations to global trends. Smaller windows (such as Scale 1 and Scale 2) are able to effectively capture the main local dynamics in the original sequence. Furthermore, the local linear layer supports complex weight initialization schemes, making feature extraction more in-depth and detailed.

[0120] The above experimental results show that the local linear layer proposed in this invention can significantly reduce the number of parameters in high-dimensional scenarios (for example, the number of parameters in the ECL dataset is only 0.00093 times that of the convolutional layer) in multi-scale time series feature extraction. At the same time, the present invention significantly improves prediction accuracy (for example, the MSE of the Solar-Energy dataset is reduced from 0.216 to 0.192). In the long-term prediction task across nine datasets such as electricity and meteorology, its prediction error (MSE / MAE) is better than that of standard convolution and pooling methods, especially in high-dimensional data scenarios, showing higher parameter efficiency and robustness.

[0121] The working principle and process of the present invention are as follows: channel-independent normalization of the input time series; setting a multi-scale time window; constraining local connections in the time dimension through a mask matrix at a single scale (for example, only retaining 15 time-step connections when the window size is 15); independently extracting the features of each variable using channel-shared linear weights at a single scale; stacking the initial scale and multi-scale sequence features into a time series feature set, and finally outputting it to a subsequent model (such as a Transformer or linear model) to complete the prediction task.

[0122] The beneficial effects of this invention include: proposing a time series feature extraction method that combines parameter efficiency with multi-scale modeling capabilities. By achieving local connections in the time dimension and sharing weights in the channel dimension (reducing the number of parameters in high-dimensional scenarios by over 99% compared to convolutional layers), the complexity of high-dimensional time series modeling is significantly reduced. Through multi-scale feature windows, local dynamics and global trends are effectively captured, improving long-term prediction accuracy. Its versatility allows integration into various time series models, providing efficient feature representation for tasks such as time series prediction.

[0123] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A method based on local linear layer (Local Linear Layer, represented as L 3 ), wherein the "local linear layer" is an artificial neural network layer proposed by the present invention in which the output neurons are linearly associated with the local input neurons. The present invention is characterized in that: The following steps are involved: S1: Standardization: Standardize the input univariate or multivariate time series; S2: Multi-scale window construction: set multiple time windows of different sizes, each window size corresponds to a scale; S3: Local feature extraction in the temporal dimension: At a single scale, a masking mechanism is used within the temporal dimension to preserve the local linear connection structure, and local features are independently extracted for each time step in each window within that scale through linear transformation. S4: Channel dimension weight sharing: In a single scale, linear weights are shared within the channel dimension, and the same linear weight matrix is ​​used for feature extraction for all variables; S5: Multi-scale feature stacking and output: The initial sequence and the feature sequences extracted at each scale are stacked along the scale dimension to form a multi-scale feature sequence for each variable, which is output for use by subsequent models. Wherein, step S1 includes the following sub-steps: S11. Time series of initial acquisition Each channel of is normalized individually: where μ (i) ,σ (i) is the mean and standard deviation of the i-th variable; is the initial sequence and standardized sequence of the i-th variable; is a set of real numbers, L is the sequence length, and C is the number of variables, also called the number of channels; S12. Introducing channel-specific trainable affine parameters: where γ i is the scaling factor, β i are translation factors, which are updated as affine parameters during model training. Wherein, step S2 includes the following sub-steps: S21. Set the window size set W = [w1, w2, ..., w G-1 ], where w t is the window size of the tth scale, G is the total number of scales, including the initial sequence. The window size increases sequentially to gradually aggregate information on larger scales; S22, each scale t (t ≥ 1) is associated with an independent trainable weight matrix and bias where L′=L+w t -1 represents the length of the sequence after padding. Wherein, step S3 includes the following sub-steps: S31, the window size w of the t-th scale t , calculate the filling length For the sequence x of variable i (i) Perform a symmetric fill operation: in is the padded sequence at the t-th scale; S32. Define the mask matrix M t ∈{0,1} L′×L , where the mask vector of the kth column is for: Where l represents the lth time step, k∈[0,N] represents the index of the output neuron, and N is the number of output neurons, that is, the number of windows at a single scale. Take S=1, the output length remains equal to the input length; S33, weight matrix V t Apply a mask: Where ⊙ is the corresponding multiplication of elements at the same position in two matrices of the same shape and size; S34, filling sequence Perform a linear change: in is the feature sequence at the t-th scale. Wherein, step S4 includes the following sub-steps: S41. When the input is a univariate sequence, that is, there is only one channel, the parameters of the current scale are directly used for processing: S42. When the input is a multivariate sequence, all variables i=1,2,…,C share a set of parameters Perform a linear transformation on each variable independently: LinearTransform(·) refers to the linear transformation in S34. Parameter sharing reduces computational complexity and accommodates inter-variable correlations. The operations of S3 and S4 on a single scale essentially pass the sequence through a local linear layer of a specific window size. Wherein, step S5 includes the following sub-steps: S51. Stack the features of the initial sequence scale (t=0) and other scales (t≥1) into a multi-scale representation: where x (i) is the initial sequence, is the characteristic sequence of the t-th scale; S52. Output the multi-scale feature set of all variables: Each Contains the multi-scale feature sequence of variable i; S53, will x w Input to subsequent models to support time series forecasting or other related tasks.

Images