Non-stationary remolding Crossform time sequence prediction method

Abstract

The invention provides a non-stationary remolding Crossform time sequence prediction method, which comprises the following steps: limiting hyper-parameters of a prediction model, taking a multivariate time sequence as the input of the prediction model, and calculating a first non-stationary factor and a second non-stationary factor through MLP and multivariate time sequence normalization; a segmented multivariate time sequence is obtained through compiling, and a first segment is a second non-stationary factor; a first segmented multivariate time sequence is obtained through one-dimensional ResNet calculation, the first segmented multivariate time sequence serves as three-layer attention input calculation to obtain a first attention multivariate time sequence, and a first coding multivariate time sequence is obtained through linear mapping; and performing linear transformation on the first segment second non-stationary factor to obtain a second segment second non-stationary factor, performing segment sequence combination on the first segment multivariate time sequence to obtain a second segment multivariate time sequence, similarly obtaining all subsequent Encoder outputs, and performing addition and reverse normalization to obtain a predicted value. According to the method, the correlation between the time sequences is considered, and the multivariate time sequence prediction precision can be improved.

Description

Technical Field

[0001] The invention belongs to the technical field of multivariate time series prediction, and specifically discloses a non-stationary reshaped Crossformer time series prediction method.

[0002] Background

[0003] With the rapid development of artificial intelligence, the demand for multi-domain prediction tasks is also increasing. For example, the power system needs to predict this year's power load based on previous years' data, the industrial sector needs to adjust process production in time according to production forecasts, the transportation sector needs to predict the passenger flow trend of the next month based on recent passenger flow, and the solid waste incineration plant needs to intervene in real time according to the parameter change trend of process variables. It can be seen that the prediction tasks in various fields help them to make plans in advance, which is conducive to improving work efficiency and economic benefits, and helps to reduce risks and economic losses. In recent years, with the in-depth study of neural networks, time series prediction models have also developed rapidly. Existing time series prediction models can be roughly divided into attention networks, convolutional networks, graph neural networks and other types. In previous work, the focus of multivariate time series prediction tasks was more on how to build a good neural network model to achieve deeper data feature extraction, but ignored the original characteristics of the retained data, which greatly affected the prediction accuracy of the model.

[0004] According to the characteristics of real data, there are two main problems in the prediction of multivariate time series: (1) Real data has non-stationary characteristics, but in most time series prediction models, excessive normalization of data reduces its non-stationarity. Using normalized data for feature extraction or calculation cannot well reflect the authenticity and accuracy of the model prediction results. (2) For multivariate time series prediction tasks, most existing models focus more on capturing the change cycle of a single time series, while ignoring the mutual influence between multivariate time series. However, in actual prediction tasks, there is a certain correlation between multivariate parameters, that is, the change of one variable often affects the change of other variables, and vice versa. Therefore, in the multivariate time series prediction task, how to solve these two problems at the same time to improve the accuracy and authenticity of the model prediction is an important issue that needs to be solved. Combined with the existing technology, the Non-Stationary Crossformer model solves this problem well by integrating non-stationary attention with the Crossformer network, but the Non-Stationary Crossformer model has shortcomings in terms of network structure and time series feature capture method. There is an urgent need to explore a multivariate time series prediction method that is computationally simple, can better capture data features, enhance the correlation calculation between multivariate time series, and restore the non-stationarity of time series. Summary of the invention

[0005] In order to solve the problem that the correlation between multivariate time series and the stationary characteristics of real data are not fully considered in the existing multivariate time series prediction, resulting in low accuracy and authenticity of the prediction results, a non-stationary reshaped Crossformer time series prediction method is proposed.

[0006] The present invention provides a non-stationary reshaped Crossformer time series prediction method, comprising the following steps:

[0007] S1. Use the hyperparameter restriction module to restrict the hyperparameters of the prediction model, taking the multivariate time series X = {x1, x2, x3, … x C},X∈R B×L×C As sample data input into the forecasting model for time series forecasting, x k represents the kth time series, k∈(1,2,3…C), R represents a real number set, B represents a sample batch, L represents the sample data step size, and C represents the time series dimension;

[0008] S2. Normalize the multivariate time series X to obtain a normalized multivariate time series X Nor , standard deviation σ x and mean μ x ;

[0009] S3. The multivariate time series X, the standard deviation σ obtained in step S2 x and the mean μ x Input the MLP network to calculate the first non-stationary factor θ and the second non-stationary factor

[0010] S4. Use the Embedding operation to perform the normalized multivariate time series X obtained in step S2. Nor and the second non-stationary factor calculated in step S3 Perform segmentation processing to obtain the segmented multivariate time series X emb and the first segment second non-stationary factor

[0011] S5. Construct a one-dimensional ResNet network to transform the segmented multivariate time series X obtained in step S4 into emb Input the one-dimensional ResNet network, and obtain the first segmented multivariate time series X after convolution and pooling. enc1 ;

[0012] S6. The first segmented multivariate time series X calculated in step S5 is enc1, the first segment second non-stationary factor obtained in step S4 The first non-stationary factor θ calculated in step S3 is used as the input of the first Encoder layer, and the three-layer attention output first attention multivariate time series enc1 of the first Encoder layer is obtained by sequentially performing non-stationary attention calculation, related attention calculation and self-attention calculation, and the obtained first attention multivariate time series is linearly mapped to obtain the output first encoding multivariate time series enc′1 of the first Encoder layer;

[0013] S7. The second non-stationary factor of the first segment obtained in step S4 Perform linear transformation to obtain the second non-stationary factor of the second segment The second non-stationary factor for the second segment Perform linear transformation to obtain the second non-stationary factor of the third segment After iterative linear transformation, the second non-stationary factor of the jth segment is obtained Where 2≤j≤E, E is the number of Encoder layers;

[0014] S8. The first segmented multivariate time series X calculated in step S5 is enc1 Merge the segmented sequences to obtain the second segmented multivariate time series X enc2 , for the second segmented multivariate time series X enc2 Merge the segmented sequences to obtain the third segmented multivariate time series X enc3 , then for the j-th Encoder layer, the j-1th segmented multivariate time series X input by the j-1th Encoder layer enc(j-1) Merge the segmented sequences to obtain the j-th segmented multivariate time series X encj ;

[0015] S9. The j-th segmented multivariate time series X calculated in step S8 is encj , the second non-stationary factor of the j-th segment calculated in step S7 The first non-stationary factor θ calculated in step S3 is used as the input of the j-th Encoder layer, and the three-layer attention output of the j-th Encoder layer is obtained through non-stationary attention calculation, related attention calculation and self-attention calculation. j , linearly map the obtained j-th attention multivariate time series to obtain the j-th encoded multivariate time series enc′ output by the j-th Encoder layer j , calculate the E-th encoded multivariate time series enc′ E , get all the encoded multivariate time series;

[0016] S10. Add and sum all the encoded multivariate time series calculated in step S9 to obtain the initial prediction value enc sum , for the initial prediction value enc sum Perform denormalization to obtain the predicted value Where T represents the prediction output step size.

[0017] According to a non-stationary reshaped Crossformer time series prediction method in some embodiments of the present application, in step S1, a hyperparameter restriction module is used to restrict the hyperparameters of the prediction model, wherein the hyperparameters include: the sample data step size L, the sequence segmentation step size S, the segmented sequence merging window size W and the number of Encoder layers E.

[0018] According to a non-stationary reshaped Crossformer time series prediction method in some embodiments of the present application, in step S1, limiting the hyperparameters of the prediction model by using a hyperparameter limiting module includes:

[0019] S110. Determine whether the sample data step length L is an integer multiple of the sequence segmentation step length S;

[0020] S120. If not, a prompt "the value of L is too small or the value of S is too large" is given, and the model training is terminated;

[0021] S130. If yes, segment the sample data according to the sequence segmentation step S, and calculate the number of segmentation segments of the sample data using formula (1).

[0022] N=L / S(1)

[0023] Formula (2) is used to calculate the remainder of the number of segmented segments N of the sample data and the merging window W, that is, the number of padded data segments required in the merging process.

[0024] N pad =MOD(N,W)(2).

[0025] According to a non-stationary reshaped Crossformer time series prediction method in some embodiments of the present application, i is set to represent the number of Encoder layers to which the hyperparameter restriction is executed, 1≤i≤E, then in the step S1, the use of the hyperparameter restriction also includes:

[0026] S140. In the process of merging the segmented sequences inputted by the i-th Encoder layer, the segmented sequences are merged according to the merging window W, and the number of the padding data segments required in the merging process is calculated using formula (2), and the number of the padding data segments N is determined. padIs it 0?

[0027] S141. If N pad If it is 0, no data padding is required, and the number of segments N of the i-th Encoder layer is updated according to formula (3).

[0028] N = N / W (3);

[0029] S142. Determine whether N is equal to 1;

[0030] S143. If N is not equal to 1, return to step S140;

[0031] S144. If N is equal to 1, determine whether i is less than E;

[0032] S145. If i is less than E, the model training of this layer is terminated, and a prompt "the value of E is too large" is given;

[0033] S146. If i is not less than E, it indicates that all Encoder layers have been traversed and the hyperparameter restriction program execution is completed.

[0034] According to a non-stationary reshaped Crossformer time series prediction method in some embodiments of the present application, in step S1, the use of hyperparameter restrictions also includes:

[0035] S150. If N pad If it is not 0, it means that the time series needs to be padded to meet the merging requirements. The number of padded data segments N is calculated according to formula (4). pad Update,

[0036] N pad =WN pad (4);

[0037] S151. Update the number of segments of the i-th Encoder layer according to formula (5).

[0038] N=(N+N pad ) / W(5);

[0039] S152. Determine whether N is equal to 1;

[0040] S153. If N is equal to 1, determine whether i is less than E;

[0041] S154. If i is less than E, the model training of this layer is terminated, and a prompt "the value of E is too large" is given;

[0042] S155. If i is not less than E, it indicates that all Encoder layers have been traversed and the hyperparameter restriction program execution ends.

[0043] According to a non-stationary reshaped Crossformer time series prediction method in some embodiments of the present application, the construction of the one-dimensional ResNet network in step S5 includes:

[0044] S510. Construct a convolution layer, wherein the convolution layer is a convolution layer with a convolution kernel of 7x7, a step size of 1, and an edge data padding of 3, and convert the segmented multivariate time series X emb Input the convolution layer to obtain the convolution multivariate time series X R ;

[0045] S520. Construct a maximum pooling layer, wherein the maximum pooling layer includes a maximum pooling layer with a convolution kernel of 5x5, a step size of 1, and an edge data padding of 2, and convolutes the multivariate time series X R Input the maximum pooling layer to obtain the maximum pooling multivariate time series X Maxp ;

[0046] S530. Construct a local feature extraction layer, and extract the maximum pooled multivariate time series X output in step S520. Maxp Input three consecutive local feature extraction layers to obtain the feature multivariate time series X out .

[0047] According to a non-stationary reshaped Crossformer time series prediction method in some embodiments of the present application, the step S5 further includes:

[0048] S540. Obtain the first segmented multivariate time series X according to formula (6) enc1 ,

[0049] X enc1 =X Maxp +X out (6).

[0050] According to a non-stationary reshaped Crossformer time series prediction method of some embodiments of the present application, the local feature extraction layer includes: 1 convolution layer with a convolution kernel of 3x3, a step size of 1, and edge data padding of 1, 1 normalization layer and a Relu activation function.

[0051] According to a non-stationary reshaped Crossformer time series prediction method in some embodiments of the present application, the calculation formula of the first attention multivariate time series enc1 in step S6 is as follows (7),

[0052]

[0053] Among them, Threeattn represents three-layer attention calculation;

[0054] Then the calculation formula of the first encoded multivariate time series enc′1 in step S6 is as follows (8):

[0055] enc′1=Projector(enc1) (8)

[0056] Among them, Projector represents linear mapping;

[0057] Then the j-th segmented multivariate time series X in step S8 encj The calculation formula is as follows (9),

[0058] X encj =merge(X enc(j-1) ,W),2≤j≤E(9)

[0059] Among them, merge represents the segmented sequence merging of sample data;

[0060] Then the j-th attention multivariate time series enc in step S9 j The calculation formula is as follows (10),

[0061]

[0062] Then the j-th encoded multivariate time series enc′ in step S9 j The calculation formula is as follows (11),

[0063] enc′ j =Projector(enc j ),2≤j≤E (11)

[0064] The encoder layer outputs the sum of the initial prediction values ​​enc sum The calculation formula is as follows (12),

[0065]

[0066] The predicted value output by the prediction model The calculation formula is as follows (13),

[0067]

[0068] Here, Denorm stands for denormalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A schematic flow chart of the non-stationary remodeling Crossformer time series prediction method provided in Example 1 of the present invention;

[0070] Figure 2A schematic diagram of the NS-RE-Crossformer model structure provided in Example 2 of the present invention;

[0071] Figure 3 A schematic diagram of a one-dimensional ResNet structure provided in Example 3 of the present invention;

[0072] Figure 4 It is a visualization diagram of the prediction results of the parameter "the right flue gas temperature of a combustion chamber" in the municipal solid waste incineration data set of the present invention;

[0073] Figure 5 This is a visualization diagram of the prediction results of the parameter “HULL” in the public data set ETTm2 of the present invention. DETAILED DESCRIPTION

[0074] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0075] Example 1: This example provides a non-stationary reshaped Crossformer time series prediction method, which can limit and optimize the hyperparameters of the time series prediction model to improve the accuracy of the prediction results. Figure 1 As shown, the following steps are included:

[0076] S1. Use the hyperparameter restriction module to restrict the hyperparameters of the prediction model, taking the multivariate time series X = {x1, x2, x3, … x C},X∈R B×L×C As sample data input into the forecasting model for time series forecasting, x k represents the kth time series, k∈(1,2,3…C), R represents a real number set, B represents a sample batch, L represents the sample data step size, and c represents the time series dimension.

[0077] S2. Normalize the multivariate time series X to obtain the normalized multivariate time series X Nor , standard deviation σ x and mean μ x .

[0078] S3. Substitute the multivariate time series X and the standard deviation σ obtained in step S2. x and mean μ x Input the MLP network to calculate the first non-stationary factor θ and the second non-stationary factor

[0079] S4. Use the Embedding operation to normalize the multivariate time series X obtained in step S2 Nor and the second non-stationary factor calculated in step S3 Perform segmentation processing to obtain the segmented multivariate time series X emb and the first segment second non-stationary factor

[0080] S5. Construct a one-dimensional ResNet network to transform the segmented multivariate time series X obtained in step S4 emb Input the one-dimensional ResNet network, and after convolution and pooling, the first segmented multivariate time series X is obtained. enc1 .

[0081] S6. The first segmented multivariate time series X calculated in step S5 is enc1 , the first segment second non-stationary factor obtained in step S4 The first non-stationary factor θ calculated in step S3 is used as the input of the first Encoder layer, and the three-layer attention output of the first Encoder layer, the first attention multivariate time series enc1, is obtained through non-stationary attention calculation, related attention calculation and self-attention calculation in sequence. The obtained first attention multivariate time series is linearly mapped to obtain the output first encoded multivariate time series enc′1 of the first Encoder layer.

[0082] S7. The second non-stationary factor of the first segment obtained in step S4 Perform linear transformation to obtain the second non-stationary factor of the second segment The second non-stationary factor of the second segment Perform linear transformation to obtain the second non-stationary factor of the third segment After iterative linear transformation, the second non-stationary factor of the jth segment is obtained Where 2≤j≤E, E is the number of Encoder layers.

[0083] S8. The first segmented multivariate time series X calculated in step S5 is enc1 Merge the segmented sequences to obtain the second segmented multivariate time series X enc2 , for the second segmented multivariate time series X enc2 Merge the segmented sequences to obtain the third segmented multivariate time series X enc3 , then for the j-th Encoder layer, the j-1th segmented multivariate time series X input by the j-1th Encoder layer enc(j-1) Merge the segmented sequences to obtain the j-th segmented multivariate time series X encj .

[0084] S9. The j-th segmented multivariate time series X calculated in step S8 is encj , the second non-stationary factor of the j-th segment calculated in step S7 The first non-stationary factor θ calculated in step S3 is used as the input of the j-th Encoder layer, and the three-layer attention output of the j-th Encoder layer is obtained by non-stationary attention calculation, related attention and self-attention calculation. j And linearly map the three-layer attention output of the j-th layer to obtain the encoded multivariate time series enc′ j , calculate the E-th encoded multivariate time series enc′ E , get all the encoded multivariate time series.

[0085] S10. Add and sum all the encoded multivariate time series calculated in step S9 to obtain the initial prediction value enc sum , for the initial prediction value enc sum Perform denormalization to obtain the predicted value Where T represents the prediction output step size.

[0086] In this embodiment, in step S1, a hyperparameter restriction module is used to restrict the hyperparameters of the prediction model, wherein the hyperparameters include: sample data step size L, sequence segmentation step size S, segment sequence merging window size W, and Encoder layer number E, etc.

[0087] This embodiment proposes a general hyperparameter restriction framework for segmented prediction models, which limits the relevant hyperparameters of the prediction model to a normal range. Then, a one-dimensional ResNet is built to extract features from the compiled data to improve the learning efficiency of the model. Finally, the Encoder architecture is optimized and adjusted, and a three-layer attention mechanism is proposed to greatly improve the accuracy of correlation calculation between multivariate time series.

[0088] This embodiment makes full use of Figure 2 The NS-RE-Crossformer model structure shown in the figure restricts the sample data length, the number of encoder layers and other hyperparameters by adding a hyperparameter restriction module before training the prediction model. During the training process, the prediction model takes the multivariate time series X as input, where X = {x1, x2, x3, … x C},X∈R B×L×C , B represents the sample batch, L represents the time series historical data step, and C represents the time series dimension. First, X is normalized to obtain X Nor , standard deviation σ x and mean μ x Next, with X, standard deviation σ x and mean μ x As the input of the MLP network, the first non-stationary factor θ and the second non-stationary factor are calculated Through the Embedding operation, X Nor and the second non-stationary factor Perform segmentation processing to obtain the segmented multivariate time series X emb and the first segment second non-stationary factor Then, with the compiled X emb As the input of the one-dimensional ResNet network, convolution, pooling and other data processing are performed to obtain the first segmented multivariate time series X enc1 Next, use X enc1 , and θ are used as the input of the first Encoder layer. After the non-stationary attention calculation, the related attention calculation and the self-attention calculation, the three-layer attention output of the first Encoder layer is obtained, the first attention multivariate time series enc1 is obtained. The obtained first attention multivariate time series is linearly mapped to obtain the output of the first Encoder layer, the first encoding multivariate time series enc′1. Then, for X enc1 , Sequence merging and linear transformation are performed to obtain the second segmented multivariate time series X enc2 , the second non-stationary factor of the second segment Then X enc2 , and θ are used as the input of the second Encoder layer. After the non-stationary attention calculation, the related attention calculation and the self-attention calculation, the three-layer attention output of the second Encoder layer is obtained, the second attention multivariate time series enc2 is obtained. The obtained second attention multivariate time series is linearly mapped to obtain the output of the second Encoder layer, the second encoded multivariate time series enc′2. Similarly, the outputs of other Encoder layers can be obtained, and the sum is added to obtain the predicted value enc sum Finally, the enc is denormalized by the anti-normalization module. sum Processing to obtain the final prediction value of the prediction model Where T represents the prediction output step size.

[0089] Example 2: This example provides a non-stationary reshaped Crossformer time series prediction method, where the multivariate time series X = {x1, x2, x3, ... x C},X∈R B×L×C As sample data input into the forecasting model for time series forecasting, x k represents the kth time series, k∈(1,2,3…C), R represents a real number set, B represents a sample batch, L represents the sample data step size, and C represents the time series dimension. The method includes the following steps:

[0090] S110. Determine whether the sample data step length L is an integer multiple of the sequence segmentation step length S.

[0091] S120. If not, it will prompt “the value of L is too small or the value of S is too large” and terminate the model training.

[0092] S130. If yes, segment the sample data according to the sequence segmentation step S, and calculate the number of segmentation segments of the sample data using formula (1).

[0093] N=L / S(1)

[0094] Formula (2) is used to calculate the remainder of the number of segmented segments N of the sample data and the merging window W, that is, the number of padded data segments required in the merging process.

[0095] N pad =MOD(N,W)(2).

[0096] Set i to represent the number of Encoder layers to which the hyperparameter restriction is applied, 1≤i≤E. Then in step S1, using the hyperparameter restriction also includes:

[0097] S140. In the process of merging the segmented sequences inputted by the i-th Encoder layer, the segmented sequences are merged according to the merging window W, and the number of padding data segments required in the merging process is calculated using formula (2), and the number of padding data segments N is determined. pad Is it 0?

[0098] S141. If N pad If it is 0, no data padding is required, and the number of segments N of the i-th Encoder layer is updated according to formula (3).

[0099] N = N / W (3);

[0100] S142. Determine whether N is equal to 1;

[0101] S143. If N is not equal to 1, return to step S140;

[0102] S144. If N is equal to 1, determine whether i is less than E;

[0103] S145. If i is less than E, the model training of this layer is terminated, and a prompt "the value of E is too large" is given;

[0104] S146. If i is not less than E, it indicates that all Encoder layers have been traversed and the hyperparameter restriction program execution is completed.

[0105] S150. If N pad If it is not 0, it means that the time series needs to be padded to meet the merging requirements. The number of padded data segments N is calculated according to formula (4).pad Update,

[0106] N pad =WN pad (4);

[0107] S151. Update the number of segments of the i-th Encoder layer according to formula (5).

[0108] N=(N+N pad ) / W(5);

[0109] S152. Determine whether N is equal to 1;

[0110] S153. If N is equal to 1, determine whether i is less than E;

[0111] S154. If i is less than E, the model training of this layer is terminated, and a prompt "the value of E is too large" is given;

[0112] S155. If i is not less than E, it indicates that all Encoder layers have been traversed and the hyperparameter restriction program execution ends.

[0113] S2. Normalize the multivariate time series X to obtain the normalized multivariate time series X Nor , standard deviation σ x and mean μ x .

[0114] S3. Substitute the multivariate time series X and the standard deviation σ obtained in step S2. x and mean μ x Input the MLP network to calculate the first non-stationary factor θ and the second non-stationary factor

[0115] S4. Use the Embedding operation to normalize the multivariate time series X obtained in step S2 Nor and the second non-stationary factor calculated in step S3 Perform segmentation processing to obtain the segmented multivariate time series X emb and the first segment second non-stationary factor

[0116] S5. Construct a one-dimensional ResNet network to transform the segmented multivariate time series X obtained in step S4 emb Input the one-dimensional ResNet network, and after convolution and pooling, the first segmented multivariate time series X is obtained. enc1 .

[0117] S6. The first segmented multivariate time series X calculated in step S5 is enc1 , the first segment second non-stationary factor obtained in step S4 The first non-stationary factor θ calculated in step S3 is used as the input of the first Encoder layer. After non-stationary attention calculation, related attention calculation and self-attention calculation, the three-layer attention output of the first Encoder layer is obtained, and the first attention multivariate time series is linearly mapped to obtain the output first encoded multivariate time series enc′1 of the first Encoder layer.

[0118] S7. The second non-stationary factor of the first segment obtained in step S4 Perform linear transformation to obtain the second non-stationary factor of the second segment The second non-stationary factor of the second segment Perform linear transformation to obtain the second non-stationary factor of the third segment After iterative linear transformation, the second non-stationary factor of the jth segment is obtained Where 2≤j≤E, E is the number of Encoder layers.

[0119] S8. The first segmented multivariate time series X calculated in step S5 is enc1 Merge the segmented sequences to obtain the second segmented multivariate time series X enc2 , for the second segmented multivariate time series X enc2 Merge the segmented sequences to obtain the third segmented multivariate time series X enc3 , then for the j-th Encoder layer, the j-1th segmented multivariate time series X input by the j-1th Encoder layer enc(j-1) Merge the segmented sequences to obtain the j-th segmented multivariate time series X encj .

[0120] S9. The j-th segmented multivariate time series X calculated in step S8 is encj , the second non-stationary factor of the jth segment calculated in step S7 The first non-stationary factor θ calculated in step S3 is used as the input of the j-th Encoder layer, and the three-layer attention output of the j-th Encoder layer is obtained through non-stationary attention calculation, related attention and self-attention calculation. j And linearly map the three-layer attention output of the j-th layer to obtain the encoded multivariate time series enc′ j , calculate the E-th encoded multivariate time series enc′ E , get all the encoded multivariate time series.

[0121] S10. Add and sum all the encoded multivariate time series calculated in step S9 to obtain the initial prediction value enc sum, for the initial prediction value enc sum Perform denormalization to obtain the predicted value Where T represents the prediction output step size.

[0122] This embodiment proposes a hyperparameter restriction module by analyzing the relationship between the hyperparameters of the prediction model and the training process, and restricts the relevant hyperparameters involved in time series segmentation and window merging to the correct range, as shown in the algorithm in Table 1.

[0123] Table 1 Hyperparameter restriction algorithm

[0124]

[0125] First, make sure that the sample data step size L is an integer multiple of the sequence segmentation step size S. If L cannot be divided by S, the sample data will not be evenly divided, which indicates that there may be two problems: the sample data step size L is too small to meet the segmentation requirements. The sequence segmentation step size S is too large, which makes it impossible to use S to divide L into integer segments. Therefore, it is necessary to issue an error warning "The value of L is too small or the value of S is too large" and terminate the program. Then, since the segmented time series needs to be merged according to the merging window W during the prediction model training process, it is necessary to calculate the number of sample data segments N = L / S, and the number of padding data segments N required during the merging process. pad =MOD(N,W).

[0126] Furthermore, this embodiment makes a judgment based on the calculation from the 1st layer Encoder to the Eth layer Encoder: first, during the segment sequence merging process, it is checked whether the calculated number of padding data segments is 0.

[0127] If the number of filled data segments is 0, it means that no data filling is required, and the number of sample data segments is updated by N = N / W. Then it is determined whether the updated N is equal to 1 and whether the last Encoder layer has been executed. If N is 1 and the last Encoder layer has not been executed, it means that the current layer has not enough data segments and cannot effectively process data, and the program execution is terminated.

[0128] If the number of padded data segments is not 0, it means that there is insufficient data in the segmented time series during the merging process, and the time series needs to be padded to meet the merging requirements. pad =WN pad , N=(N+N pad) / W updates the number of segments N. Then, it checks whether the updated N is equal to 1 and whether the last Encoder layer has been executed. If N is 1 and the last Encoder layer has not been executed, it means that the current layer has not enough data segments and cannot effectively process data, so the program execution is terminated.

[0129] Example 3: This example provides a non-stationary reshaped Crossformer time series prediction method, comprising the following steps:

[0130] S1. Use the hyperparameter restriction module to restrict the hyperparameters of the prediction model, taking the multivariate time series X = {x1, x2, x3, … x C},X∈R B×L×C As sample data input into the forecasting model for time series forecasting, x k represents the kth time series, k∈(1,2,3…C), R represents a real number set, B represents a sample batch, L represents the sample data step size, and C represents the time series dimension.

[0131] S2. Normalize the multivariate time series X to obtain the normalized multivariate time series X Nor , standard deviation σ x and mean μ x .

[0132] S3. Substitute the multivariate time series X and the standard deviation σ obtained in step S2. x and mean μ x Input the MLP network to calculate the first non-stationary factor θ and the second non-stationary factor

[0133] S4. Use the Embedding operation to normalize the multivariate time series X obtained in step S2 Nor and the second non-stationary factor calculated in step S3 Perform segmentation processing to obtain the segmented multivariate time series X emb and the first segment second non-stationary factor

[0134] S510. Construct a convolutional layer with a convolution kernel of 7x7, a step size of 1, and an edge data padding of 3. emb Input the convolution layer to get the convolution multivariate time series X R ;

[0135] S520. Construct a maximum pooling layer, which includes a maximum pooling layer with a convolution kernel of 5x5, a step size of 1, and an edge data padding of 2. Convolve the multivariate time series X R Input the maximum pooling layer to obtain the maximum pooling multivariate time series XMaxp ;

[0136] S530. Construct a local feature extraction layer to extract the maximum pooled multivariate time series X output from step S520. Maxp Input 3 consecutive local feature extraction layers to obtain the feature multivariate time series X out .

[0137] S540. Obtain the first segmented multivariate time series X according to formula (6) enc1 ,

[0138] X enc1 =X Maxp +X out (6).

[0139] S6. The first segmented multivariate time series X calculated in step S540 is enc1 , the first segment second non-stationary factor obtained in step S4 The first non-stationary factor θ calculated in step S3 is used as the input of the first Encoder layer. After non-stationary attention calculation, related attention calculation and self-attention calculation, the three-layer attention output of the first Encoder layer is obtained, and the first attention multivariate time series is linearly mapped to obtain the output first encoded multivariate time series enc′1 of the first Encoder layer.

[0140] S7. The second non-stationary factor of the first segment obtained in step S4 Perform linear transformation to obtain the second non-stationary factor of the second segment The second non-stationary factor of the second segment Perform linear transformation to obtain the second non-stationary factor of the third segment After iterative linear transformation, the second non-stationary factor of the jth segment is obtained Where 2≤j≤E, E is the number of Encoder layers.

[0141] S8. The first segmented multivariate time series X calculated in step S540 is enc1 Merge the segmented sequences to obtain the second segmented multivariate time series X enc2 , for the second segmented multivariate time series X enc2 Merge the segmented sequences to obtain the third segmented multivariate time series X enc3 , then for the j-th Encoder layer, the j-1th segmented multivariate time series X input by the j-1th Encoder layer enc(j-1) Merge the segmented sequences to obtain the j-th segmented multivariate time series X encj .

[0142] S9. The j-th segmented multivariate time series X calculated in step S8 is encj , the second non-stationary factor of the jth segment calculated in step S7 The first non-stationary factor θ calculated in step S3 is used as the input of the j-th Encoder layer, and the three-layer attention output of the j-th Encoder layer is obtained through non-stationary attention calculation, related attention and self-attention calculation. j And linearly map the three-layer attention output of the j-th layer to obtain the encoded multivariate time series enc′ j , calculate the E-th encoded multivariate time series enc′ E , get all the encoded multivariate time series.

[0143] S10. Add and sum all the decoded multivariate time series calculated in step S9 to obtain enc sum , for enc sum Perform denormalization to obtain the predicted value Where T represents the prediction output step size.

[0144] In order to better capture data features without affecting the performance of the prediction model, this embodiment builds Figure 3 The one-dimensional ResNet shown in the figure adds a convolutional network to reduce data noise at the beginning of training and extract useful local features. In the network architecture of the one-dimensional ResNet, a convolutional layer with a convolution kernel of 7x7, a stride of 1, and an edge data padding of 3 is first constructed to capture the global features of the input data. Next, a maximum pooling layer with a convolution kernel of 5x5, a stride of 1, and an edge data padding of 2 is used to further extract important local features and reduce the dimension of the data. Subsequently, local feature extraction is performed through three convolutional layers with a convolution kernel of 3x3, a stride of 1, and an edge data padding of 1. These convolutional layers can capture local patterns in the data in more detail. At the same time, in order to accelerate the training process of the prediction model, a batch normalization layer is added after each convolutional layer to standardize the input of each layer, thereby accelerating the convergence speed of the prediction model and improving the stability of the prediction model. The addition of the Relu activation function enhances the nonlinear expression ability of the prediction model and alleviates the problem of gradient disappearance. In the final stage of the network, the output X of the maximum pooling layer is added. Maxp And the output X after multi-layer convolutional layer processing out Add to get the input X of the first Encoder layer enc1 ,This residual connection helps to reduce the learning complexity of the prediction model and promotes faster convergence of the prediction model. In this way, the network can learn and optimize more effectively, thereby improving the overall performance.

[0145] Preferably, the local feature extraction layer in this embodiment includes: 1 convolution layer with a convolution kernel of 3x3, a step size of 1, and edge data padding of 1, 1 normalization layer and a Relu activation function.

[0146] Preferably, the calculation formula of the first attention multivariate time series enc1 in step S6 of this embodiment is as follows (7):

[0147]

[0148] Among them, Threeattn represents three-layer attention calculation;

[0149] Then the calculation formula of the first encoded multivariate time series enc′1 in step S6 is as follows (8):

[0150] enc′1=Projector(enc1) (8)

[0151] Among them, Projector represents linear mapping;

[0152] Then the j-th segmented multivariate time series X in step S8 encj The calculation formula is as follows (9),

[0153] X encj =merge(X enc(j-1) ,W),2≤j≤E(9)

[0154] Among them, merge represents the segmented sequence merging of sample data;

[0155] Then the j-th attention multivariate time series enc in step S9 j The calculation formula is as follows (10),

[0156]

[0157] Then the j-th encoded multivariate time series enc′ in step S9 j The calculation formula is as follows (11),

[0158] enc′ j =Projector(enc j ),2≤j≤E (11)

[0159] The encoder layer outputs the sum of the initial prediction values ​​enc sum The calculation formula is as follows (12),

[0160]

[0161] The predicted value output by the prediction model The calculation formula is as follows (13),

[0162]

[0163] Here, Denorm stands for denormalization.

[0164] The embodiment of the present invention mainly uses the multivariate time series data set of municipal solid waste incineration to test and verify the performance of the prediction model, and also uses 4 public multivariate time series data sets to expand the test and verify the performance of the prediction model:

[0165] Municipal Solid Waste Incineration (MSWI) Dataset: This dataset mainly includes the left, middle, and right flue gas temperatures of a combustion chamber collected in one day, the average temperature of a combustion chamber, the air flow on the left side of the drying grate, the air flow on the right side of the combustion section grate, the cumulative supply flow of urea solvent, the cumulative amount of urea solution, the cumulative amount of activated carbon storage bin feed, the total amount of economizer feed water, the steam temperature at the outlet of the third-stage superheater, the flue gas oxygen concentration, and the flue gas NOx concentration data.

[0166] Public datasets:

[0167] Dataset 1: Electricity, this dataset contains hourly electricity consumption data of 321 customers collected from 2012 to 2014.

[0168] Dataset 2: Weather, this dataset includes 21 weather indicators in 2020, with data recorded every 10 minutes.

[0169] Dataset 3: Exchange_rate, which contains daily exchange rate data for eight countries from 1990 to 2016.

[0170] Dataset 4: ETTm2, which contains 15-minute interval data from July 2016 to July 2018, covering the load characteristics of seven types of oil and power transformers.

[0171] The present invention selects five multivariate time series prediction models for comparative experiments, as follows:

[0172] Model 1: The Non-stationary Transformer model proposes to restore the non-stationary characteristics of the time series by calculating the non-stationary factors, thereby improving the prediction accuracy of the model.

[0173] Model 2: The Transformer model, with its core self-attention mechanism, effectively models and learns sequence data through techniques such as multi-head attention, position encoding, and residual connection, making it a key model in fields such as natural language processing.

[0174] Model 3: Crossformer is a novel neural network model based on the Transformer architecture. It introduces a two-layer attention mechanism to better capture global and local information, and performs well in tasks such as image classification and object detection.

[0175] Model 4: Informer introduces a multi-granularity temporal attention mechanism to effectively process long-sequence temporal data. It is able to simultaneously handle long-term, medium-term, and short-term temporal dependencies in a single training.

[0176] Model 5: Non-stationary Crossformer mainly integrates non-stationary attention into Crossformer to simultaneously achieve non-stationary restoration of time series and correlation calculation.

[0177] The specific experimental process is as follows:

[0178] The embodiments of the present invention use PyTorch to perform all experiments on an Intel i7 processor and an NVIDIA GeForce RTX 3070 GPU. The historical data length of all prediction models is set to 168, and the prediction length is T∈{32,48,96,168}. During the experiment, in addition to the solid waste incineration dataset, seven variables were randomly selected from other public datasets containing multiple parameters for comparison. The dataset is divided into a training set (70%), a validation set (10%), and a test set (20%). The initial learning rate of each model is set to 10 -4 , the sample batch size is 32. The number of encoder layers is set to 4, and the training cycle is set to 20. The performance of the prediction model is evaluated by the mean square error (MSE) and the mean absolute error (MAE). The smaller the error, the better the performance. The error formula is as follows.

[0179]

[0180] In the above formula, M is the total number of samples, Y is i is the actual value of the sample, is the predicted value of the sample.

[0181] As shown in Table 2, the NS-RE-Crossformer model constructed by the embodiment of the present invention performs well in the prediction tasks of different lengths of 5 data sets. Among the 40 comparison results, 32 results of the prediction model ranked first and 8 results ranked second. These rankings are determined based on the minimum values ​​of the mean square error (MSE) and the mean absolute error (MAE).

[0182] like Figure 4As shown, in order to highlight the prediction performance of the model, this embodiment selects the "flue gas temperature on the right side of a combustion chamber" parameter from the municipal solid waste incineration data set for visualization.

[0183] Table 2. Experimental results of prediction model error comparison

[0184]

[0185]

[0186] The ablation study of the embodiment of the present invention is as follows. The non-stationary reshaped Crossformer time series prediction method provided in this embodiment mainly includes three parts: 1) hyperparameter restriction; 2) one-dimensional ResNet; 3) encoder structure adjustment (ESA). Since the hyperparameter restriction is intended to standardize the input of the hyperparameters of the prediction model, this embodiment mainly conducts ablation experiments on the latter two parts. This embodiment uses the Non-stationary Crossformer model as the basic framework and conducts prediction experiments of different lengths on the ETTm2 dataset. This embodiment uses mean square error (MSE) and mean absolute error (MAE) to evaluate the performance, as shown in Table 3. In addition, in order to better visualize the performance of different prediction models, this embodiment selects the "HULL" parameter from the ETTm2 dataset for visualization, as shown in Figure 5 As shown. The experimental results show that the non-stationary reshaped Crossformer time series prediction method proposed in this embodiment can gradually improve the performance of the prediction model in different step-size prediction tasks. At the same time, the prediction model combining the three methods shows the best prediction performance.

[0187] Table 3. Comparison results of prediction model errors in ablation experiments

[0188]

[0189] The embodiment of the present invention studies the non-stationary nature of time series, the correlation between multivariate time series, and the data feature capture based on the Non-Stationary Crossformer model. In most existing multivariate time series prediction models, the focus is on exploring the interactions between time series or trying to restore the original features of the data, without considering these two aspects at the same time. Although the Non-Stationary Crossformer model considers these two points at the same time, it still has defects in terms of model structure and data feature capture. Based on this, the embodiment of the present invention proposes the NS-RE-Crossformer model. By adding hyperparameter restrictions, one-dimensional ResNet and ESA, the shortcomings of the original model are solved, and the prediction accuracy of the prediction model and the efficiency of time series correlation calculation are improved. Through the above optimization, the prediction model provided by the embodiment of the present invention shows excellent performance in the prediction experiments of 5 real data sets.

[0190] The embodiments of the present invention are given for the purpose of illustration and description, and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. A non-stationary reshaped Crossformer time series prediction method, characterized in that: The steps include: S1. Use the hyperparameter restriction module to restrict the hyperparameters of the prediction model, taking the multivariate time series X = {x1, x2, x3, … x C },X∈R B×L×C As sample data input into the forecasting model for time series forecasting, x k represents the kth time series, k∈(1,2,3…C), R represents a real number set, B represents a sample batch, L represents the sample data step size, and C represents the time series dimension; S2. Normalize the multivariate time series X to obtain a normalized multivariate time series X Nor , standard deviation σ x and mean μ x ; S3. The multivariate time series X, the standard deviation σ obtained in step S2 x and the mean μ x Input the MLP network to calculate the first non-stationary factor θ and the second non-stationary factor S4. Use the Embedding operation to perform the normalized multivariate time series X obtained in step S2. Nor and the second non-stationary factor calculated in step S3 Perform segmentation processing to obtain the segmented multivariate time series X emb and the first segment second non-stationary factor S5. Construct a one-dimensional ResNet network to transform the segmented multivariate time series X obtained in step S4 into emb Input the one-dimensional ResNet network, and obtain the first segmented multivariate time series X after convolution and pooling. enc1 ; S6. The first segmented multivariate time series X calculated in step S5 is enc1 , the first segment second non-stationary factor obtained in step S4 The first non-stationary factor θ calculated in step S3 is used as the input of the first Encoder layer, and the three-layer attention output first attention multivariate time series enc1 of the first Encoder layer is obtained by non-stationary attention calculation, related attention calculation and self-attention calculation in sequence, and the obtained first attention multivariate time series is linearly mapped to obtain the output first encoding multivariate time series enc1′ of the first Encoder layer; S7. The second non-stationary factor of the first segment obtained in step S4 Perform linear transformation to obtain the second non-stationary factor of the second segment The second non-stationary factor for the second segment Perform linear transformation to obtain the second non-stationary factor of the third segment After iterative linear transformation, the second non-stationary factor of the jth segment is obtained Where 2≤j≤E, E is the number of Encoder layers; S8. The first segmented multivariate time series X calculated in step S5 is enc1 Merge the segmented sequences to obtain the second segmented multivariate time series X enc2 , for the second segmented multivariate time series X enc2 Merge the segmented sequences to obtain the third segmented multivariate time series X enc3 , then for the j-th Encoder layer, the j-1th segmented multivariate time series X input by the j-1th Encoder layer enc(j-1) Merge the segmented sequences to obtain the j-th segmented multivariate time series X encj ; S9. The j-th segmented multivariate time series X calculated in step S8 is encj , the second non-stationary factor of the j-th segment calculated in step S7 The first non-stationary factor θ calculated in step S3 is used as the input of the j-th Encoder layer, and the three-layer attention output of the j-th Encoder layer is obtained by non-stationary attention calculation, related attention and self-attention calculation. j And linearly map the three-layer attention output of the jth layer to obtain the encoded multivariate time series enc j ′, and then calculate the E-th encoded multivariate time series enc′ E , get all the encoded multivariate time series; S10. Add and sum all the encoded multivariate time series calculated in step S9 to obtain the initial prediction value enc sum , for the initial prediction value enc sum Perform denormalization to obtain the predicted value Where T represents the prediction output step size.

2. According to claim 1, a non-stationary reshaped Crossformer time series prediction method is characterized in that: In the step S1, a hyperparameter restriction module is used to restrict the hyperparameters of the prediction model, wherein the hyperparameters include: the sample data step size L, the sequence segmentation step size S, the segmented sequence merging window size W and the number of encoder layers E.

3. The non-stationary remodeling Crossformer time series prediction method according to claim 2, characterized in that: In step S1, limiting the hyperparameters of the prediction model by using the hyperparameter limiting module includes: S110. Determine whether the sample data step length L is an integer multiple of the sequence segmentation step length S; S120. If not, it will prompt "the value of L is too small or the value of S is too large", and terminate the model training; S130. If yes, segment the sample data according to the sequence segmentation step S, and calculate the number of segmentation segments of the sample data using formula (1). N=L / S(1) Formula (2) is used to calculate the remainder of the number of segmented segments N of the sample data and the merging window W, that is, the number of padded data segments required in the merging process. N pad =MOD(N,W)(2)。 4. The non-stationary reshaped Crossformer time series prediction method according to claim 3 is characterized in that: Assume that i represents the number of Encoder layers to which the hyperparameter restriction is applied, 1≤i≤E. Then in step S1, the use of the hyperparameter restriction further includes: S140. In the process of merging the segmented sequences inputted by the i-th Encoder layer, the segmented sequences are merged according to the merging window W, and the number of the padding data segments required in the merging process is calculated using formula (2), and the number of the padding data segments N is determined. pad Is it 0? S141. If N pad If it is 0, no data padding is required, and the number of segments N of the i-th Encoder layer is updated according to formula (3). N = N / W (3); S142. Determine whether N is equal to 1; S143. If N is not equal to 1, return to step S140; S144. If N is equal to 1, determine whether i is less than E; S145. If i is less than E, the model training of this layer is terminated, and a prompt "the value of E is too large" is given; S146. If i is not less than E, it indicates that all Encoder layers have been traversed and the hyperparameter restriction program execution is completed.

5. The non-stationary reshaped Crossformer time series prediction method according to claim 4 is characterized in that: In step S1, the use of hyperparameter restrictions also includes: S150. If N pad If it is not 0, it means that the time series needs to be padded to meet the merging requirements. The number of padded data segments N is calculated according to formula (4). pad Update, N pad =W-N pad (4); S151. Update the number of segments of the i-th Encoder layer according to formula (5). N=(N+N pad ) / W(5); S152. Determine whether N is equal to 1; S153. If N is equal to 1, determine whether i is less than E; S154. If i is less than E, the model training of this layer is terminated, and a prompt "the value of E is too large" is given; S155. If i is not less than E, it indicates that all Encoder layers have been traversed and the hyperparameter restriction program execution ends.

6. The non-stationary reshaped Crossformer time series prediction method according to claim 1, characterized in that: The construction of the one-dimensional ResNet network in step S5 includes: S510. Construct a convolution layer, wherein the convolution layer is a convolution layer with a convolution kernel of 7x7, a step size of 1, and an edge data padding of 3, and convert the segmented multivariate time series X emb Input the convolution layer to obtain the convolution multivariate time series X R ; S520. Construct a maximum pooling layer, wherein the maximum pooling layer includes a maximum pooling layer with a convolution kernel of 5x5, a step size of 1, and an edge data padding of 2, and convolutes the multivariate time series X R Input the maximum pooling layer to obtain the maximum pooling multivariate time series X Maxp ; S530. Construct a local feature extraction layer, and extract the maximum pooled multivariate time series X output in step S520. Maxp Input three consecutive local feature extraction layers to obtain the feature multivariate time series X out .

7. The non-stationary reshaped Crossformer time series prediction method according to claim 6, characterized in that: The step S5 further includes: S540. Obtain the first segmented multivariate time series X according to formula (6) enc1 , X enc1 =X Maxp +X out (6)。 8. The non-stationary reshaped Crossformer time series prediction method according to claim 7, characterized in that: The local feature extraction layer includes: a convolution layer with a convolution kernel of 3x3, a step size of 1, and edge data padding of 1, a normalization layer and a Relu activation function.

9. The non-stationary reshaped Crossformer time series prediction method according to claim 1, characterized in that: The calculation formula of the first attention multivariate time series enc1 in step S6 is as follows (7): Among them, Threeattn represents three-layer attention calculation; Then the calculation formula of the first encoded multivariate time series enc1′ in step S6 is as follows (8): enc′1=Projector(enc1) (8) Among them, Projector represents linear mapping; Then the j-th segmented multivariate time series X in step S8 encj The calculation formula is as follows (9), X encj =merge(X enc(j-1) ,W),2≤j≤E (9)where, merge represents merging the segmented sequences of sample data; Then the j-th attention multivariate time series enc in step S9 j The calculation formula is as follows (10), Then the j-th encoded multivariate time series enc′ in step S9 j The calculation formula is as follows (11), enc′ j =Projector(enc j ),2≤j≤E (11) The encoder layer outputs the sum of the initial prediction values ​​enc sum The calculation formula is as follows (12), The predicted value output by the prediction model The calculation formula is as follows (13), Here, Denorm stands for denormalization.

Images