A method and system for cross-variable long-term time series forecasting that integrates linear and enhanced Transformers

Abstract

This invention provides a method and system for cross-variable long-term time series prediction that integrates linear and enhanced Transformer methods. The method includes: extracting an input sequence using a sliding window based on historical data and calculating the temporal statistical features of the sequence; generating scaling and bias factors for the window through a shared fully connected network to perform affine normalization on the input and eliminate distribution bias; performing a Fast Fourier Transform on the normalized tensor to extract the main frequency domain components and map them back to the time domain, then fusing positional encoding to form a time-frequency hybrid embedding; constructing an adjacency matrix based on the correlation between variables, using this matrix as a mask in cross-variable attention calculation to suppress interference between weakly correlated variables and enhance variable co-representation; mapping the representation to the prediction length through a time projection layer, and performing an inverse transform using previously generated normalization parameters to restore the original dimensions of the data and obtain the prediction result.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, and in particular relates to a method and system for cross-variable long-term time series forecasting that integrates linear and enhanced Transformer. Background Technology

[0002] Existing multivariate or multivariate long-term time series prediction (LTSF) methods are mainly divided into two categories: Transformer-based and linear / lightweight methods. Representative works in the Transformer approach include Informer, which reduces the overhead of long sequences with sparse attention; Autoformer, which introduces sequence decomposition and autocorrelation mechanisms to enhance long-term prediction; FEDformer, which combines decomposition and frequency domain representation to obtain global profiles and improve efficiency; and PatchTST, which improves performance and computational efficiency through patching and channel independence strategies. While these methods have improved the ability to model long sequences, most schemes still focus on optimizing "time dependence," and cross-variable dependencies are often implicitly learned. When there are lag couplings, scale differences, or large noise between variables, the expression of cross-variable relationships may be unstable. At the same time, some Transformer structures are complex, resulting in high training / inference costs and significant engineering deployment pressure. Moreover, methods based on the Transformer architecture have the ability to model long-range dependencies. However, real-world time series data are non-stationary, meaning that the statistical distribution of the data shifts over time, leading to inconsistencies between the distribution of training and testing data, which affects the model's generalization ability. Furthermore, in multivariate time series, there are correlations among variables, and the contributions of different variables to the prediction target vary greatly. Current long-term time series forecasting techniques fail to detect dependencies between variables or uncover potential periodic patterns from multiple time-frequency domains. The RevIN method only uses statistics for linear transformation, which can easily lead to the loss of some key trend information. When dealing with dependencies between variables, conventional cross-variable attention mechanisms calculate the attention score of fully connected pairs for all variables, which incurs computational costs and introduces noise interference from low-correlation variables, making it difficult for the model to focus on the synergistic effects of variables. At the same time, most methods ignore the supplementary value of periodic components in the frequency domain for long-term trend prediction. Therefore, there is an urgent need for a long-term time series forecasting method that can synergistically optimize the handling of non-stationary distributions, the fusion of frequency domain features, and the selection of highly correlated variables to improve the accuracy of predictions. Summary of the Invention

[0003] To address the problem that existing technologies cannot synergistically optimize long-term time series forecasting methods that handle non-stationary distributions, fuse frequency domain features, and screen highly correlated variables, this invention proposes, in a first aspect, a cross-variable long-term time series forecasting method that integrates linear and enhanced Transformer methods, comprising: Historical observation data of multivariate time series are obtained, and the input subsequence is truncated according to the set window length. The statistical features of the input subsequence in the time dimension are calculated, and the mapping relationship between the statistical features and the affine transformation parameters is learned using a fully connected network with shared weights. Scaling factors and bias factors are generated, and the input subsequence is normalized and subjected to element-wise affine transformation using the scaling factors and bias factors to obtain a normalized tensor. A fast Fourier transform is performed on the normalized tensor to extract the k largest frequency components in the frequency domain. The components are then converted back to time domain features through a frequency-time mapping module and fused with the time domain position code to obtain a hybrid embedding vector containing time and frequency information. A variable adjacency matrix is ​​constructed based on the correlation between variables in historical data. The hybrid embedding vector is input into the cross-variable attention module. When calculating the multi-head attention score, the variable adjacency matrix is ​​used to generate an attention mask. The dot product result of the query vector and the key vector is corrected by the mask. The value vector is aggregated based on the corrected attention matrix to obtain the variable collaborative representation. The time dimension of the variable co-representation is mapped to the prediction window length using a time projection layer to obtain a normalized prediction result. Based on the inverse operation of the scaling factor and bias factor and the statistical characteristics of the input subsequence, the normalized prediction result is denormalized to restore the original dimensions of the data and obtain the time series prediction result.

[0004] In another aspect, the present invention also proposes a cross-variable long-term time series forecasting system that integrates linear and enhanced Transformer methods, comprising the following modules: The affine module is used to acquire historical observation data of multivariate time series, truncate the input subsequence according to a set window length, calculate the statistical features of the input subsequence in the time dimension, learn the mapping relationship between the statistical features and the affine transformation parameters using a fully connected network with shared weights, generate scaling factors and bias factors, and perform normalization and element-wise affine transformation on the input subsequence using the scaling factors and bias factors to obtain a normalized tensor. The fusion module is used to perform a fast Fourier transform on the normalized tensor, extract the k largest frequency components in the frequency domain, convert the components back to time domain features through the frequency-time mapping module, and fuse them with the time domain position code to obtain a hybrid embedding vector containing time and frequency information. The aggregation module is used to construct a variable adjacency matrix based on the correlation between variables in historical data. The hybrid embedding vector is input into the cross-variable attention module. When calculating the multi-head attention score, the variable adjacency matrix is ​​used to generate an attention mask. The dot product result of the query vector and the key vector is masked and corrected. The value vector is aggregated based on the corrected attention matrix to obtain the variable collaborative representation. The recovery module is used to map the time dimension of the variable co-representation to the prediction window length using the time projection layer to obtain the normalized prediction result. Based on the inverse operation of the scaling factor and the bias factor and the statistical characteristics of the input subsequence, the normalized prediction result is denormalized to restore the original dimensions of the data and obtain the time series prediction result.

[0005] This invention constructs a fully connected network with shared weights to learn the mapping relationship between statistical features and transformation parameters, generating scaling and bias factors for the input subsequences. Normalization and affine transformations are then performed at the input, eliminating distribution shift issues in non-stationary time series. By combining frequency domain component extraction and time domain location encoding, a hybrid feature representation incorporating time-frequency information is constructed, enhancing the model's ability to perceive the inherent periodicity and trends of the data. A mask generated using an adjacency matrix based on variable correlation is used to correct the dot product result when calculating cross-variable attention, filtering out noise interference from low-correlation variables and solving the problem of redundant irrelevant information in multivariate prediction. Finally, an inverse normalization strategy restores the original dimensions of the data, improving the stability of long-term time series prediction while maintaining computational efficiency. Attached Figure Description

[0006] Figure 1 A flowchart of the first embodiment; Figure 2 This is a schematic diagram of Top-k filtering in the frequency domain; Figure 3 This is a schematic diagram comparing the errors in the ablation experiment. Detailed Implementation

[0007] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0008] See Figure 1 The flowchart of the first embodiment shown specifically includes: S1. Obtain historical observation data of multivariate time series, and cut the input subsequence according to the set window length; calculate the statistical features of the input subsequence in the time dimension, use a fully connected network with shared weights to learn the mapping relationship between statistical features and affine transformation parameters, generate scaling factors and bias factors, and use the scaling factors and bias factors to perform normalization and element-wise affine transformation on the input subsequence to obtain a normalized tensor and eliminate non-stationary distribution bias. Specifically, a time series dataset containing N variables is collected, the length of the historical observation window is set to L, the length of the prediction window is set to T, and the sliding window technique is used to slice the multivariate time series data. For any time step t, the continuous data from time t-L+1 to time t is extracted as the input subsequence, and an input matrix of dimension L×N is constructed, where L represents the time step and N represents the total number of variables.

[0009] In one embodiment, specifically for power load forecasting, the dataset is the power transformer temperature and load dataset ETTh1, sourced from the power transformer operation monitoring system, with a sampling interval of 1 hour. Variables include multiple operational indicators such as oil temperature (OT), high-voltage load (HUFL), high-voltage reactive load (HULL), medium-voltage load (MUFL), medium-voltage reactive load (MULL), low-voltage load (LUFL), and low-voltage reactive load (LULL). The measured values ​​of these variables at consecutive time points are recorded in chronological order. For example, when the input window length L=96, a continuous 96-hour observation matrix containing 7 variables can be extracted as an input subsequence, represented as a 96×7 real number matrix, where each row corresponds to the operating status for one hour, and each column corresponds to a specific power operation indicator.

[0010] For an input matrix of dimension L×N, the mean and standard deviation of each variable are calculated along the time dimension L, resulting in a mean vector and a standard deviation vector of dimension 1×N. The mean and standard deviation vectors are concatenated along the feature channel dimension and input into a parameter generation network consisting of two linear layers and an activation function. This network outputs a scaling factor γ and a bias factor β of dimension 1×N. The input matrix is ​​then standardized using the mean and standard deviation vectors (i.e., the input matrix is ​​subtracted from the mean vector and divided by the standard deviation vector) to obtain a standardized matrix. This standardized matrix is ​​then multiplied element-wise by the scaling factor γ and the bias factor β is added to obtain a normalized tensor after affine transformation. Calculating the mean and standard deviation of the input subsequence along the time dimension during the normalization stage eliminates the amplitude drift caused by seasonal and operational variations in power load.

[0011] More specifically, the step of acquiring historical observation data of multivariate time series and truncating the input subsequence according to a set window length includes: The collected multivariate time series data were divided into training set, validation set and test set. The sliding window algorithm was used to process each dataset. The input window length was set to L. Data segments with shape L rows and N columns were continuously extracted from the dataset in chronological order as input subsequences, where L is a positive integer and N is the total number of variables.

[0012] In the data preprocessing stage, the original multivariate time series is divided into training, validation, and test sets in a 7:1:2 ratio to ensure the objectivity of model evaluation. In the sliding window processing, the historical observation window length L of the input sequence is set to 96, and the prediction window length T is also 96. For a dataset containing N variables, such as N=7, a batch input tensor of dimension (B, 96, 7) is generated through a sliding operation with a step size of 1 time unit, where B is the batch size. This constructs continuous samples containing local temporal dependencies.

[0013] In an optional embodiment, the calculation of the statistical features of the input subsequence in the time dimension utilizes a fully connected network with shared weights to learn the mapping relationship between the statistical features and the affine transformation parameters, generating scaling factors and bias factors, including: Calculate the mean vector and standard deviation vector of the input subsequence in the time dimension respectively, and concatenate them to construct a statistical feature vector with a dimension twice the total number of variables; The statistical feature vector is input into a two-layer multilayer perceptron. The first layer maps the features to the hidden dimension, and the second layer maps the features back to a dimension with twice the total number of variables. The first half of the elements of the output vector are used as scaling factors, and the second half are used as bias factors.

[0014] The multilayer perceptron described is a feedforward neural network model. Structurally, it includes an input layer, hidden layers, and an output layer. The number of neurons in the input layer is set to twice the total number of variables, i.e., 2N. The number of neurons in the hidden layer is set to 64, and the number of neurons in the output layer is set to 2N. An activation function connects the hidden and output layers. The training set uses multivariate time-series training data as described above. During training, the network is treated as part of an overall prediction model. The loss function between the predicted results and the true labels is calculated, and the network weight parameters are updated using the backpropagation algorithm.

[0015] Calculate the input subsequence Mean over time and standard deviation The two are then concatenated to obtain a statistical feature vector of length 2N. This vector is then input into a two-layer MLP network containing activation functions: the first linear layer maps the dimension from 2N to the hidden layer dimension, for example... After processing with ReLU or GELU activation functions, the second linear layer remaps the dimension from 64 back to 2N. The output vector is split into two parts, and the first N elements after Sigmoid activation are used as scaling factors. The last N elements are used as bias factors. This allows for the generation of unique distribution parameters for each sliding window.

[0016] To eliminate the non-stationarity of the input sequence and prevent division by zero errors, in an optional embodiment, the normalization and element-wise affine transformation of the input subsequence using the scaling factor and bias factor to obtain a normalized tensor includes: The standardized data is obtained by subtracting the mean vector of the corresponding variable from each element in the input subsequence and dividing by the sum of the standard deviation vector of the corresponding variable and the smoothing constant. The standardized data is multiplied element-wise by the scaling factor, and the product is added element-wise by the bias factor to generate a normalized tensor with distribution alignment.

[0017] Z-score standardization is performed using statistical characteristics, and a smoothing constant is used. The calculation formula is: This operation forces the data distribution across different time windows to align to a latent space with zero mean and unit variance. A learnable affine transformation is then applied. Generate a normalized tensor with distribution alignment, where This represents element-wise multiplication, using the parameters generated above. and The recovery of inherent non-stationary features in the data that are beneficial to the prediction task eliminates the covariate drift problem caused by distribution shift while preserving the unique pattern information of the sequence.

[0018] S2, Perform a fast Fourier transform on the normalized tensor, extract the k largest frequency components in the frequency domain, convert the components back to time domain features through the frequency-time mapping module, and fuse them with the time domain position code to obtain a hybrid embedding vector containing time and frequency information. Specifically, a one-dimensional fast Fourier transform is performed on each column of the normalized tensor to obtain a complex representation in the frequency domain. The amplitude of all frequency components is calculated. Based on the set hyperparameter k, such as k=L / 4 or k=L / 2, the indices of the k frequency components with the largest amplitudes are selected to construct a frequency-preserving mask matrix. Insignificant frequency components are set to zero. The processed frequency domain features are restored to the time domain signal through an inverse fast Fourier transform. A linear projection layer is used to map the restored time domain signal to a high-dimensional feature space, resulting in a time-domain feature tensor of dimension L×N×d, where d is the dimension of the model's hidden layer. Simultaneously, a learnable position encoding matrix of dimension L×d is generated. This matrix is ​​then broadcast-expanded to L×N×d and added element-wise to the time-domain feature tensor to obtain a hybrid embedding vector.

[0019] In a preferred embodiment, the frequency components are determined based on changes in power load. Specifically, since the sampling interval of the ETTh1 dataset is 1 hour, the identification of frequency components is based on the mapping relationship between the frequency domain index k and the period T of the discrete Fourier transform, k=L / T, where L is the length of the input window. The frequency index corresponding to the daily load cycle is calculated to be approximately L / 24, and the frequency index corresponding to the weekly load cycle is approximately L / 168. The two dominant frequencies and their harmonics that are concentrated in the amplitude energy in the spectrum are retained as the core heat accumulation characteristics of the transformer dominated by the industrial and commercial power consumption patterns. Other non-periodic high-frequency components are treated as random noise or transient disturbances and filtered to zero, thereby extracting the long-term trend of the driving oil temperature change in the frequency domain.

[0020] In an optional embodiment, the step of extracting the k largest frequency components in the frequency domain and converting the components back to time domain features through a frequency-time mapping relationship includes: Perform a one-dimensional fast Fourier transform on each variable dimension of the normalized tensor to obtain complex frequency domain coefficients, and calculate the amplitude of all frequency components. Select the k frequency components with the largest amplitude values, and set the real and imaginary parts of the remaining frequency components to 0 to obtain sparse frequency domain features. A one-dimensional inverse fast Fourier transform is performed on the sparse frequency domain features, the real part is taken as the initial time domain features, and the feature dimension is adjusted through a linear mapping layer to output a time domain feature sequence with the same length as the input subsequence.

[0021] A real-number fast Fourier transform is used to convert the time dimension L to the frequency domain. For an input of L=96, 49 complex frequency modes are obtained. After calculating the amplitude spectrum of each mode, the k components with the largest amplitudes are selected, such as... Figure 2The complex coefficients of the remaining high-frequency noise or irrelevant components are forced to zero to achieve frequency domain sparsity. The phase information of the selected component is preserved. The signal dimension after inverse fast Fourier transform is (B,L,N). It is linearly projected through a fully connected layer, which is equivalent to a learnable global filter. It can remove noise while preserving the key periodic trends in the time series. The generated time-domain feature sequence dimension is (B,L,N,d).

[0022] S3. Construct a variable adjacency matrix based on the correlation between variables in historical data. Input the hybrid embedding vector into the cross-variable attention module. When calculating the multi-head attention score, use the variable adjacency matrix to generate an attention mask. Perform mask correction on the dot product result of the query vector and the key vector to filter out the interference of low-correlation variables. Aggregate the value vector based on the corrected attention matrix to obtain the variable collaborative representation. Specifically, the Pearson correlation coefficient between each pair of variables in the training set is calculated, and a correlation threshold is set. When the absolute value of the correlation coefficient is greater than the threshold, the corresponding position in the variable adjacency matrix is ​​set to 1; otherwise, it is set to 0, generating an N×N binary variable adjacency matrix. Based on this binary variable adjacency matrix, an attention mask is constructed. In the mask matrix, positions with an adjacency matrix value of 1 are assigned 0, and positions with an adjacency matrix value of 0 are assigned negative infinity. The dimension of the hybrid embedding vector is reshaped to N×L×d. Treating the variable dimension N as the sequence length, three linear transformation matrices are used to map the L×d-dimensional features corresponding to each variable, thereby uniformly compressing the dimension to the single-head attention dimension. The generation dimension is (B,N, The query vector Q, key vector K, and value vector V satisfy d = h × Let h be the number of heads in the multi-head attention. Calculate the dot product of Q and the transpose of K, and divide by the scaling factor to obtain the original attention score matrix. Add the attention mask as a bias term to the original attention score matrix to minimize the attention scores of low-relevance variable pairs. Perform a Softmax operation on the last dimension of the corrected score matrix to obtain normalized attention weights. Multiply these weights by the value vector V to obtain the attention output of each head. The multi-head attention concatenates the outputs of all heads along the feature dimension, maps them back to the model's hidden dimension d through a linear layer, and then restores the time dimension L through dimension transformation, finally obtaining a variable collaborative representation of dimension (B, N, L, d).

[0023] In an optional embodiment, constructing the variable adjacency matrix based on the correlation between variables in historical data includes: Calculate the Pearson correlation coefficient between any two variable sequences in the historical observation data training set, and construct the correlation coefficient matrix; Set a correlation threshold, traverse the correlation coefficient matrix, and if the absolute value of an element is greater than or equal to the correlation threshold, assign a value of 1 to the corresponding position in the variable adjacency matrix; if the absolute value of an element is less than the correlation threshold, assign a value of 0 to the corresponding position in the variable adjacency matrix, thereby generating a sparse binary variable adjacency matrix.

[0024] Before model training, the Pearson correlation coefficients between all variable pairs (i,j) in the training set are pre-calculated. This forms an N×N correlation matrix. A hyperparameter threshold is then set. The preferred range is 0.5 to 0.7, for example, 0.6, used to filter strongly correlated variables. Construct a binary adjacency matrix. ,when hour, Otherwise, it is 0. This step constructs a sparse variable dependency graph structure, which reduces the fitting overhead of invalid information in the model by masking the attention weights of low-correlation variable pairs, while avoiding noise interference from fully connected attention, ensuring that the model only focuses on highly correlated variable pairs.

[0025] In an optional embodiment, the step of generating an attention mask using a variable adjacency matrix and performing mask correction on the dot product of the query vector and the key vector includes: In the cross-variable attention module, the matrix product of the query matrix and the transpose of the key matrix is ​​calculated and divided by the scaling factor to obtain the original attention score matrix. An attention mask matrix is ​​constructed based on the binary variable adjacency matrix, wherein the mask value corresponding to the position with an adjacency matrix value of 1 is 0, and the mask value corresponding to the position with an adjacency matrix value of 0 is negative infinity. The original attention score matrix and the attention mask matrix are added element-wise to obtain the corrected correlation matrix, and then the corrected correlation matrix is ​​normalized using Softmax.

[0026] The intervariate attention module structure includes linear projection layers for generating the query matrix, key matrix, and value matrix, respectively, and an attention computation unit based on scaled dot products. The training set consists of preprocessed and embedded multivariate time series training data. The training process involves end-to-end supervised learning along with the overall model, jointly optimizing the projection layer weights and other model parameters using gradient descent. The input consists of the feature-mapped query matrix, key matrix, and value matrix, as well as a mask matrix generated based on the variable adjacency matrix. The output is a weighted aggregation of contextual features based on the correlations between variables.

[0027] In a multi-head attention mechanism, for example, with h=8 heads and d=512 feature dimensions, the calculation... Obtain the original score. Construct a mask matrix M, setting the mask value corresponding to the position with a value of 0 in the adjacency matrix to... Positions with a value of 1 are set to 0. After adding M to the original score, Softmax is applied to make the attention weights of irrelevant variables approach 0. The calculation formula is as follows: The hard threshold masking strategy forces the model to ignore low-correlation noise variables when aggregating information, thereby improving the accuracy of multivariate collaborative representation.

[0028] S4. The time dimension of the variable co-representation is mapped to the prediction window length using the time projection layer to obtain the normalized prediction result. Based on the inverse operation of the scaling factor and the bias factor and the statistical characteristics of the input subsequence, the normalized prediction result is denormalized to restore the original dimensions of the data and obtain the time series prediction result.

[0029] Specifically, the variable co-representation is input to the time projection layer, which is composed of a multilayer perceptron with an input dimension of L and an output dimension of the prediction window length T. Each variable channel of the co-representation is mapped, and the output dimension is a normalized prediction tensor of T×N. An inverse affine transformation is performed, first subtracting the bias factor β generated above from the normalized prediction tensor, then dividing by the scaling factor γ, multiplying the result by the sum of the standard deviation vector and the smoothing constant calculated above, and adding the mean vector to restore the original distribution of the data and obtain the predicted values ​​for the next T time steps.

[0030] In an optional embodiment, the time projection layer adopts a multilayer perceptron structure consisting of two fully connected layers. The first fully connected layer maps the input L-dimensional temporal features to a 2L-dimensional hidden feature space, and uses the GELU activation function to ensure the nonlinear fitting ability of the model, while avoiding the dead neuron and gradient vanishing problems that are prone to occur with the ReLU activation function. The second fully connected layer does not set an activation function to ensure the continuous distribution characteristics of the output predicted value, and maps the hidden layer features from 2L dimensions to a preset prediction window length T dimensions. The weight parameters of this multilayer perceptron are globally shared among all variable channels. At the same time, all weight parameters of this time projection layer can be updated end-to-end with the overall prediction model through gradient descent, realizing the mapping from historical temporal features to future multi-step prediction sequences.

[0031] To control the transformer cooling system, the normalized output value is restored to the actual Celsius value through inverse affine transformation or a self-network containing a fully connected layer. The input of the sub-network is the time series prediction result, and the output is the oil temperature change sequence, including the oil temperature change curve for the next 96 to 720 hours. This prediction curve is compared with the safety threshold of the transformer insulating oil, such as 85°C or 90°C. If the predicted oil temperature exceeds the threshold at some future time, an early warning signal is output to guide the dispatch center to reduce the load distribution of the transformer or start the forced oil cooling device in advance, thereby avoiding accelerated insulation aging or tripping accidents caused by excessive temperature.

[0032] In an optional embodiment, the step of performing denormalization on the normalized prediction result to restore the original dimensions of the data and obtain the time series prediction result includes: Obtain the normalized prediction sequence output by the time projection layer, and subtract the bias factor from the sequence element by element to obtain the intermediate result; Divide the intermediate result element by element by the scaling factor to obtain the deaffine result; The deaffine result is multiplied element by element by the sum of the standard deviation vector of the input subsequence and the smoothing constant, and then the mean vector of the input subsequence is added to obtain the time series prediction result.

[0033] Using the bias factor generated above and scaling factor Output of the model Perform deaffine transformation: , This is element-wise division. The mean value is calculated using the input window. and standard deviation Perform anti-standardization: , This is an element-wise multiplication. The process maps the prediction results back from the stationary latent space to the dimensional range of the original data, ensuring the effectiveness of the loss function calculation and making the output prediction sequence usable for practical business decisions.

[0034] Based on the multivariate time series prediction model proposed in this invention, the following three ablation experiments are planned to verify the effectiveness of the core module.

[0035] Experiment 1: Ablation Experiment of Affine Transformation Module While keeping other structures unchanged, the mapping module that uses a fully connected network to learn statistical features to affine transformation parameters is removed. Specifically, the Z-score normalization process is retained, but the scaling factor is forced to be set to an all-1 vector and the bias factor to an all-0 vector. That is, only fixed statistics are used for normalization and denormalization, without performing learnable affine transformations.

[0036] Specific experimental data: The power load dataset ETTh1 was selected as the experimental subject.

[0037] Set the input sequence length L to 96 and the prediction window length T to 96.

[0038] Set the batch size to 32 and the number of training iterations to 10.

[0039] The mean squared error (MSE) of the complete model on the test set is 0.375, and the mean absolute error (MAE) is 0.398.

[0040] After removing the affine transformation module, the mean squared error (MSE) of the test set was 0.412, and the mean absolute error (MAE) was 0.425.

[0041] Compared to the complete model, removing the affine transformation module increased the mean squared error (MSE) by 0.037 and the mean absolute error (MAE) by 0.027.

[0042] Experiment 2: Ablation Experiment of Frequency Domain Sparse Feature Extraction Module While keeping other structures unchanged, the frequency domain feature extraction module is removed. Specifically, the normalized tensor is mapped to features through a linear layer in the time dimension, without performing Fast Fourier Transform, amplitude filtering, or Inverse Fourier Transform operations, thus completing feature processing in the time domain.

[0043] The power load dataset ETTh1 was selected as the experimental subject.

[0044] Set the input sequence length L to 96 and the prediction window length T to 96.

[0045] The number of frequency components k retained in the frequency domain is set to 5.

[0046] The mean squared error (MSE) of the complete model on the test set is 0.375, and the mean absolute error (MAE) is 0.398.

[0047] The mean squared error (MSE) of the test set after removing the frequency domain sparse feature extraction module was 0.405, and the mean absolute error (MAE) was 0.419.

[0048] Compared to the complete model, removing the frequency domain sparse feature extraction module increased the mean squared error (MSE) by 0.030 and the mean absolute error (MAE) by 0.021.

[0049] Experiment 3: Ablation Experiment of Variable Correlation Masking Mechanism While keeping other structures unchanged, the mechanism for generating attention masks based on the variable adjacency matrix is ​​removed. Specifically, in the cross-variable attention module, the Pearson correlation coefficient is not calculated, nor is a sparse binary variable adjacency matrix constructed. Instead, fully connected attention calculations are allowed between all variables, meaning that the attention score matrix is ​​not superimposed with any mask correction.

[0050] The power load dataset ETTh1 was selected as the experimental subject.

[0051] Set the input sequence length L to 96 and the prediction window length T to 96.

[0052] The correlation threshold is set to 0.6.

[0053] The mean squared error (MSE) of the complete model on the test set is 0.375, and the mean absolute error (MAE) is 0.398.

[0054] After removing the variable correlation masking mechanism, the mean squared error (MSE) of the test set is 0.392, and the mean absolute error (MAE) is 0.410.

[0055] Compared to the complete model, removing the variable correlation masking mechanism increased the mean squared error (MSE) by 0.017 and the mean absolute error (MAE) by 0.012. Figure 3 .

[0056] In a second embodiment, the present invention also proposes a cross-variable long-term time series forecasting system that integrates linear and enhanced Transformer methods, comprising the following modules: The affine module is used to acquire historical observation data of multivariate time series, truncate the input subsequence according to a set window length, calculate the statistical features of the input subsequence in the time dimension, learn the mapping relationship between the statistical features and the affine transformation parameters using a fully connected network with shared weights, generate scaling factors and bias factors, and perform normalization and element-wise affine transformation on the input subsequence using the scaling factors and bias factors to obtain a normalized tensor. The fusion module is used to perform a fast Fourier transform on the normalized tensor, extract the k largest frequency components in the frequency domain, convert the components back to time domain features through the frequency-time mapping module, and fuse them with the time domain position code to obtain a hybrid embedding vector containing time and frequency information. The aggregation module is used to construct a variable adjacency matrix based on the correlation between variables in historical data. The hybrid embedding vector is input into the cross-variable attention module. When calculating the multi-head attention score, the variable adjacency matrix is ​​used to generate an attention mask. The dot product result of the query vector and the key vector is masked and corrected. The value vector is aggregated based on the corrected attention matrix to obtain the variable collaborative representation. The recovery module is used to map the time dimension of the variable co-representation to the prediction window length using the time projection layer to obtain the normalized prediction result. Based on the inverse operation of the scaling factor and the bias factor and the statistical characteristics of the input subsequence, the normalized prediction result is denormalized to restore the original dimensions of the data and obtain the time series prediction result.

[0057] In an optional embodiment, the step of acquiring historical observation data of multivariate time series and truncating the input subsequence according to a set window length includes: The collected multivariate time series data were divided into training set, validation set and test set. The sliding window algorithm was used to process each dataset. The input window length was set to L. Data segments with shape L rows and N columns were continuously extracted from the dataset in chronological order as input subsequences, where L is a positive integer and N is the total number of variables.

[0058] In an optional embodiment, the calculation of the statistical features of the input subsequence in the time dimension utilizes a fully connected network with shared weights to learn the mapping relationship between the statistical features and the affine transformation parameters, generating scaling factors and bias factors, including: Calculate the mean vector and standard deviation vector of the input subsequence in the time dimension respectively, and concatenate them to construct a statistical feature vector with a dimension twice the total number of variables; The statistical feature vector is input into a two-layer multilayer perceptron. The first layer maps the features to the hidden dimension, and the second layer maps the features back to a dimension with twice the total number of variables. The first half of the elements of the output vector are used as scaling factors, and the second half are used as bias factors.

[0059] In an optional embodiment, the step of performing normalization and element-wise affine transformation on the input subsequence using the scaling factor and bias factor to obtain a normalized tensor includes: The standardized data is obtained by subtracting the mean vector of the corresponding variable from each element in the input subsequence and dividing by the sum of the standard deviation vector of the corresponding variable and the smoothing constant. The standardized data is multiplied element-wise by the scaling factor, and the product is added element-wise by the bias factor to generate a normalized tensor with distribution alignment.

[0060] In an optional embodiment, the step of extracting the k largest frequency components in the frequency domain and converting the components back to time domain features through a frequency-time mapping module includes: Perform a one-dimensional fast Fourier transform on each variable dimension of the normalized tensor to obtain complex frequency domain coefficients, and calculate the amplitude of all frequency components. Select the k frequency components with the largest amplitude values, and set the real and imaginary parts of the remaining frequency components to 0 to obtain sparse frequency domain features. A one-dimensional inverse fast Fourier transform is performed on the sparse frequency domain features, the real part is taken as the initial time domain features, and the feature dimension is adjusted through a linear mapping layer to output a time domain feature sequence with the same length as the input subsequence.

[0061] In an optional embodiment, constructing the variable adjacency matrix based on the correlation between variables in historical data includes: Calculate the Pearson correlation coefficient between any two variable sequences in the historical observation data training set, and construct the correlation coefficient matrix; Set a correlation threshold, traverse the correlation coefficient matrix, and if the absolute value of an element is greater than or equal to the correlation threshold, assign a value of 1 to the corresponding position in the variable adjacency matrix; if the absolute value of an element is less than the correlation threshold, assign a value of 0 to the corresponding position in the variable adjacency matrix, thereby generating a sparse binary variable adjacency matrix.

[0062] In an optional embodiment, the step of generating an attention mask using a variable adjacency matrix and performing mask correction on the dot product of the query vector and the key vector includes: In the cross-variable attention module, the matrix product of the query matrix and the transpose of the key matrix is ​​calculated and divided by the scaling factor to obtain the original attention score matrix. An attention mask matrix is ​​constructed based on the binary variable adjacency matrix, wherein the mask value corresponding to the position with an adjacency matrix value of 1 is 0, and the mask value corresponding to the position with an adjacency matrix value of 0 is negative infinity. The original attention score matrix and the attention mask matrix are added element-wise to obtain the corrected correlation matrix, and then the corrected correlation matrix is ​​normalized using Softmax.

[0063] In an optional embodiment, the step of performing denormalization on the normalized prediction result to restore the original dimensions of the data and obtain the time series prediction result includes: Obtain the normalized prediction sequence output by the time projection layer, and subtract the bias factor from the sequence element by element to obtain the intermediate result; Divide the intermediate result element by element by the scaling factor to obtain the deaffine result; The deaffine result is multiplied element by element by the sum of the standard deviation vector of the input subsequence and the smoothing constant, and then the mean vector of the input subsequence is added to obtain the time series prediction result.

[0064] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0065] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for forecasting long-term cross-variable time series by fusing linear and enhanced Transformer methods, characterized in that, Includes the following steps: Historical observation data of multivariate time series are obtained, and the input subsequence is truncated according to the set window length. The statistical features of the input subsequence in the time dimension are calculated, and the mapping relationship between the statistical features and the affine transformation parameters is learned by using a fully connected network with shared weights. Scaling factors and bias factors are generated, and the input subsequence is normalized and subjected to element-wise affine transformation using the scaling factors and bias factors to obtain a normalized tensor. A fast Fourier transform is performed on the normalized tensor to extract the k largest frequency components in the frequency domain. The components are then converted back to time domain features through a frequency-time mapping module and fused with the time domain position code to obtain a hybrid embedding vector containing time and frequency information. A variable adjacency matrix is ​​constructed based on the correlation between variables in historical data. The hybrid embedding vector is input into the cross-variable attention module. When calculating the multi-head attention score, the variable adjacency matrix is ​​used to generate an attention mask. The dot product result of the query vector and the key vector is corrected by the mask. The value vector is aggregated based on the corrected attention matrix to obtain the variable collaborative representation. The time dimension of the variable co-representation is mapped to the prediction window length using a time projection layer to obtain a normalized prediction result. Based on the inverse operation of the scaling factor and bias factor and the statistical characteristics of the input subsequence, the normalized prediction result is denormalized to restore the original dimensions of the data and obtain the time series prediction result.

2. The method according to claim 1, characterized in that, The process of acquiring historical observation data of multivariate time series and truncating the input subsequence according to a set window length includes: The collected multivariate time series data were divided into training set, validation set and test set. The sliding window algorithm was used to process each dataset. The input window length was set to L. Data segments with shape L rows and N columns were continuously extracted from the dataset in chronological order as input subsequences, where L is a positive integer and N is the total number of variables.

3. The method according to claim 1, characterized in that, The calculation of the statistical features of the input subsequence in the time dimension utilizes a fully connected network with shared weights to learn the mapping relationship between the statistical features and the affine transformation parameters, generating scaling factors and bias factors, including: Calculate the mean vector and standard deviation vector of the input subsequence in the time dimension respectively, and concatenate them to construct a statistical feature vector with a dimension twice the total number of variables; The statistical feature vector is input into a two-layer multilayer perceptron. The first layer maps the features to the hidden dimension, and the second layer maps the features back to a dimension with twice the total number of variables. The first half of the elements of the output vector are used as scaling factors, and the second half are used as bias factors.

4. The method according to claim 2, characterized in that, The step of performing normalization and element-wise affine transformation on the input subsequence using the scaling factor and bias factor to obtain a normalized tensor includes: The standardized data is obtained by subtracting the mean vector of the corresponding variable from each element in the input subsequence and dividing by the sum of the standard deviation vector of the corresponding variable and the smoothing constant. The standardized data is multiplied element-wise by the scaling factor, and the product is added element-wise by the bias factor to generate a normalized tensor with distribution alignment.

5. The method according to claim 1, characterized in that, The step of extracting the k largest frequency components in the frequency domain and converting these components back to time domain features using a frequency-time mapping module includes: Perform a one-dimensional fast Fourier transform on each variable dimension of the normalized tensor to obtain complex frequency domain coefficients, and calculate the amplitude of all frequency components. Select the k frequency components with the largest amplitude values, and set the real and imaginary parts of the remaining frequency components to 0 to obtain sparse frequency domain features. A one-dimensional inverse fast Fourier transform is performed on the sparse frequency domain features, the real part is taken as the initial time domain features, and the feature dimension is adjusted through a linear mapping layer to output a time domain feature sequence with the same length as the input subsequence.

6. The method according to claim 1, characterized in that, The construction of the variable adjacency matrix based on the correlation between variables in historical data includes: Calculate the Pearson correlation coefficient between any two variable sequences in the historical observation data training set, and construct the correlation coefficient matrix; Set a correlation threshold, traverse the correlation coefficient matrix, and if the absolute value of an element is greater than or equal to the correlation threshold, assign a value of 1 to the corresponding position in the variable adjacency matrix; if the absolute value of an element is less than the correlation threshold, assign a value of 0 to the corresponding position in the variable adjacency matrix, thereby generating a sparse binary variable adjacency matrix.

7. The method according to claim 6, characterized in that, The step of generating an attention mask using a variable adjacency matrix and performing mask correction on the dot product of the query vector and the key vector includes: In the cross-variable attention module, the matrix product of the query matrix and the transpose of the key matrix is ​​calculated and divided by the scaling factor to obtain the original attention score matrix. An attention mask matrix is ​​constructed based on the binary variable adjacency matrix, wherein the mask value corresponding to the position with an adjacency matrix value of 1 is 0, and the mask value corresponding to the position with an adjacency matrix value of 0 is negative infinity. The original attention score matrix and the attention mask matrix are added element-wise to obtain the corrected correlation matrix, and then the corrected correlation matrix is ​​normalized using Softmax.

8. The method according to claim 6, characterized in that, The step of performing denormalization on the normalized prediction results to restore the original dimensions of the data and obtain the time series prediction results includes: Obtain the normalized prediction sequence output by the time projection layer, and subtract the bias factor from the sequence element by element to obtain the intermediate result; Divide the intermediate result element by element by the scaling factor to obtain the deaffine result; The deaffine result is multiplied element by element by the sum of the standard deviation vector of the input subsequence and the smoothing constant, and then the mean vector of the input subsequence is added to obtain the time series prediction result.

9. A long-term time series forecasting system integrating linear and enhanced Transformer methods, characterized in that, Includes the following modules: The affine module is used to acquire historical observation data of multivariate time series, truncate the input subsequence according to a set window length, calculate the statistical features of the input subsequence in the time dimension, learn the mapping relationship between the statistical features and the affine transformation parameters using a fully connected network with shared weights, generate scaling factors and bias factors, and perform normalization and element-wise affine transformation on the input subsequence using the scaling factors and bias factors to obtain a normalized tensor. The fusion module is used to perform a fast Fourier transform on the normalized tensor, extract the k largest frequency components in the frequency domain, convert the components back to time domain features through the frequency-time mapping module, and fuse them with the time domain position code to obtain a hybrid embedding vector containing time and frequency information. The aggregation module is used to construct a variable adjacency matrix based on the correlation between variables in historical data. The hybrid embedding vector is input into the cross-variable attention module. When calculating the multi-head attention score, the variable adjacency matrix is ​​used to generate an attention mask. The dot product result of the query vector and the key vector is masked and corrected. The value vector is aggregated based on the corrected attention matrix to obtain the variable collaborative representation. The recovery module is used to map the time dimension of the variable co-representation to the prediction window length using the time projection layer to obtain the normalized prediction result. Based on the inverse operation of the scaling factor and the bias factor and the statistical characteristics of the input subsequence, the normalized prediction result is denormalized to restore the original dimensions of the data and obtain the time series prediction result.

10. The system according to claim 9, characterized in that, The process of acquiring historical observation data of multivariate time series and truncating the input subsequence according to a set window length includes: The collected multivariate time series data were divided into training set, validation set and test set. The sliding window algorithm was used to process each dataset. The input window length was set to L. Data segments with shape L rows and N columns were continuously extracted from the dataset in chronological order as input subsequences, where L is a positive integer and N is the total number of variables.

Images