Multivariate time series prediction method based on multi-cycle semantic interaction network

Abstract

The invention discloses a multivariate time series prediction method based on a multi-cycle semantic interaction network, and the method is characterized in that the method comprises the steps: S1, collecting multivariate time series data according to a fixed time interval; s2, for the historical sequence, performing data preprocessing in a time dimension and a feature dimension according to a sequence from the feature dimension to the time dimension; s3, initializing a multi-cycle semantic interaction network; s4, training a multi-cycle semantic interaction network by using the data samples preprocessed in S1 to S2, so as to predict a feature value at a future moment according to a feature value observed in the past; the preprocessed data samples are divided into a training set, a verification set and a test set according to the proportion, and the training set, the verification set and the test set are used for training a prediction network, verifying the training effect and evaluating prediction precision. According to the method, the long-term and short-term time correlation can be fully mined and fused, the transmission efficiency of the cyclic network is enhanced, and the accuracy of multivariate time sequence prediction is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of time series prediction, and in particular to a multivariate time series prediction method based on a multi-period semantic interaction network. Background Art

[0002] With the development of deep learning, multivariate time series forecasting is becoming an increasingly effective tool for data mining and analysis, covering a wide range of industries, including energy, weather, transportation, and finance. A multivariate time series refers to multiple sequences of data sampled at fixed intervals for multiple observational features. Multivariate time series forecasting involves predicting multiple feature values at future moments based on multiple feature values from historical observations. The key to accurate forecasting lies in fully exploiting both long-term and short-term temporal correlations.

[0003] Long Short-Term Memory (LSTM) is a recurrent neural network that is well-suited for processing time series data and has achieved good performance in various time prediction scenarios. However, LSTM networks have many parameters and require long training times. To reduce these parameters, the Gated Recurrent Unit (GRU) was developed. Compared to LSTM, the GRU has fewer parameters, lower computational complexity, and faster training and inference speeds, thus optimizing the recurrent network structure to a certain extent.

[0004] However, LSTM and GRU can only take a single subsequence as input, cannot extract long-term and short-term temporal dependencies separately, and cannot fully tap into the deep semantic information contained in subsequences divided by different periods. Furthermore, LSTM and GRU can only transmit information in one direction and cannot simultaneously learn multiple periodic patterns, resulting in low computational efficiency. Summary of the Invention

[0005] The present invention aims to provide a multivariate time series prediction method based on a multi-period semantic interaction network to solve the above problems.

[0006] The technical solution of the present invention is: a multivariate time series prediction method based on a multi-period semantic interaction network, comprising:

[0007] S1, collects multivariate time series data at fixed time intervals;

[0008] S2, for historical sequences, performs data preprocessing in the time dimension and feature dimension in the order from feature dimension to time dimension;

[0009] S3, initialize the multi-cycle semantic interaction network;

[0010] S4 uses the data samples preprocessed by S1-S2 to train a multi-cycle semantic interaction network to predict the feature values of future moments based on the feature values observed in the past; the preprocessed data samples are divided into training set, validation set, and test set according to the proportion, which are used to train the prediction network, verify the training effect, and evaluate the prediction accuracy.

[0011] Preferably, in S1, the collected multivariate time series data is recorded as the original data set Where T represents the total number of sampled time steps, and N represents the number of features sampled at each time step. If the length of the historical sequence is H and the length of the predicted sequence is P, the historical sequence is recorded as The predicted sequence is recorded as

[0012] The original data set is divided into data samples according to the length of the historical sequence and the length of the predicted sequence. The length of each data sample is H+P, and the historical sequence part and the predicted sequence part are marked separately. Among them, the historical sequence part is used as the network input, including the time dependency to predict the future sequence, and the predicted sequence part is used as the true value label to evaluate the prediction results.

[0013] Preferably, in S2, in the feature dimension, outlier detection and normalization are performed on the sampling sequence of the same feature;

[0014] In the time dimension, the sampling time is re-divided into a two-dimensional grid structure according to the "sampling day" standard, that is, the time grid. Each row represents the sampling values of all time steps in the same day, and each column represents the sampling values of the same time point on different days;

[0015] The preprocessed historical data is constructed into a three-dimensional matrix, where two dimensions represent the time grid and the other dimension represents the feature sequence sampled at the corresponding moment, which is recorded as R represents the number of time grid rows, C represents the number of time grid columns, Represents the feature sequence in the rth row and cth column in the time grid.

[0016] Preferably, in S3, the multi-cycle semantic interaction network is a grid-like recurrent network, which is composed of semantic interaction units arranged according to a time grid, that is, the semantic interaction unit located in the Rth row and Cth column is responsible for processing the feature sequence located in the Rth row and Cth column in the three-dimensional matrix.

[0017] The semantic interaction unit includes long short-term memory variants and fusion gates;

[0018] Each semantic interaction unit processes the feature sequence sampled at the corresponding moment and transmits the processed data to the adjacent semantic interaction unit, using a bidirectional loop structure to simultaneously mine multi-cycle correlations.

[0019] Preferably, in S3, the multi-cycle semantic interaction network has three output vectors, which are represented by H hor , H ver , Out means:

[0020] H hor ,H ver ,Out=MPSIN(X 2D ), (1)

[0021] in, is the horizontal loop output vector, is the vertical cycle output vector, L is the number of network layers, d model is the hidden unit dimension, is the output vector of the last layer.

[0022] Preferably, in the semantic interaction unit, the long short-term memory variant includes a first long short-term memory variant and a second long short-term memory variant; wherein the first long short-term memory variant responsible for horizontal transmission is denoted as SIU hor The second long short-term memory variant responsible for longitudinal transmission is denoted as SIU ver , fusion gated o gate Responsible for fusing the horizontal transmission results with the vertical transmission results; each semantic interaction unit is responsible for processing the feature sequence sampled at the corresponding moment. The specific process is as follows:

[0023]

[0024] Where r, c, and l in the subscripts represent the rth row (1≤r≤R), the cth column (1≤c≤C), and the lth layer (1≤l≤L), respectively;

[0025] When l=1, Represents the input of the first layer. When l>1, That is, the output of the l-1 layer is used as the input of the l layer;

[0026] and They represent the horizontal input of the current unit (i.e. the output of the previous unit in the same row) and the vertical input (i.e. the output of the unit above in the same column). When r=1, Replace it with a vector of all 0s, when c = 1, Replace with a vector of all 0s;

[0027] is the output of the horizontal long-short time variant and is used as the input of the following unit in the same row; is the output of the vertical long-short time variant and is used as the input of the following unit in the same column; r,c,l is the output of the fused gate and is used as the input of the unit in the same row and column of the next layer.

[0028] Preferably, in the semantic interaction unit, the long short-term memory variant includes a main control gate and an auxiliary control gate, which respectively extract long-term and short-term temporal dependencies during the cyclic transmission process. Its working principle is as follows:

[0029]

[0030] Among them, p t Represents the master control gate, s t Represents the auxiliary control door, represents the main hidden state, Represents a secondary hidden state. is the output at time t, is the main direction input at time t, is the secondary direction input at time t, and x is the original input;

[0031] σ and tanh represent sigmoid and tanh activation functions respectively, e represents the Hardman product, is the weight parameter, is the bias parameter. Similarly, is the weight parameter, is the bias parameter;

[0032] For the first long short-term memory variant SIU responsible for lateral transmission hor For example, as shown in formula (2), the inputs are x=input r,c,l , Its output Horizontal transmission corresponds to the connection direction of all time steps on the same day in the time network. The output of the previous unit in the same row is used as the input of the primary direction, and the output of the previous unit in the same column is used as the input of the secondary direction. By distinguishing between the primary and secondary directions, the network is trained based on learning multiple aspects of information to automatically discover the periodic patterns in the corresponding transmission direction, thereby fully exploring short-term temporal correlations through horizontal circulation.

[0033] For the second long short-term memory variant SIU responsible for longitudinal transmission ver For example, as shown in formula (3), x = input r,c,l , Its output Vertical transmission corresponds to the connection direction of the same time point on different days in the time grid. The output of the upper unit in the same column is used as the input of the main direction, and the output of the previous unit in the same row is used as the input of the secondary direction. By distinguishing the main direction from the secondary direction, the network is trained on the basis of learning multiple aspects of information to automatically mine the periodic patterns in the corresponding transmission direction, so that long-term time correlation can be fully explored through vertical circulation.

[0034] Preferably, the fusion gate gate , responsible for SIU hor and SIU ver The output of is fused to obtain multi-cycle time correlation, and the working process is as follows:

[0035]

[0036] o r,c,l =o t ef t , (12)

[0037] Among them, t represents the fusion gate, f t represents the fused hidden state, is the weight parameter, is the bias parameter.

[0038] Preferably, in S4, the output vector of the last layer Connected to the prediction model composed of a multi-layer perceptron, the prediction module is responsible for mapping the intermediate results that integrate multiple time correlations into prediction results.

[0039] The beneficial effects of the present invention are:

[0040] The present invention can fully explore and integrate long-term and short-term temporal correlations, enhance the transmission efficiency of cyclic networks, and improve the accuracy of multivariate time series prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A step diagram of a multivariate time series prediction method based on a multi-period semantic interaction network provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of a time grid provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of a multi-periodic semantic interaction network in which semantic interaction units are arranged according to a time grid, provided by an embodiment of the present invention;

[0044] Figure 4 Schematic diagram of a multi-cycle semantic interaction network with three output vectors provided by an embodiment of the present invention;

[0045] Figure 5 A schematic diagram of a multi-cycle semantic interaction network including two long short-term memory variants and a fusion gate provided by an embodiment of the present invention;

[0046] Figure 6 Schematic diagram of LSTM gating and GRU provided in an embodiment of the present invention;

[0047] Figure 7 Schematic diagram of a semantic interaction unit including two gates provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand and implement the present invention. The embodiments of the present invention are not limited thereto.

[0049] Example 1

[0050] like Figure 1 As shown in Figure 2, the multivariate time series prediction method based on the multi-period semantic interaction network mainly includes:

[0051] (1) Collect multivariate time series data at fixed time intervals and record them as the original data set Where T represents the total number of sampled time steps, and N represents the number of features sampled at each time step. Assuming the length of the historical sequence is H and the length of the predicted sequence is P, then the historical sequence is recorded as The predicted sequence is recorded as The original data set is divided into data samples according to the length of the historical sequence and the length of the predicted sequence. The length of each data sample is H+P, and the historical sequence part and the predicted sequence part are marked separately. The historical sequence part is used as the network input, which contains the time dependency relationship to predict the future sequence. The predicted sequence part is used as the true value label to evaluate the prediction results.

[0052] (2) For historical sequences, data preprocessing is performed in the time dimension and feature dimension respectively in the order of “feature dimension first, then time dimension”.

[0053] In the feature dimension, the sampling sequence of the same feature is subjected to outlier detection and normalization;

[0054] like Figure 2 As shown in the figure, in the time dimension, the sampling time is re-divided into a two-dimensional grid structure according to the "sampling day" standard, which is called the time grid. Each row represents the sampling values of all time steps in the same day, and each column represents the sampling values of the same time point on different days.

[0055] It can be seen that the preprocessed historical data is constructed into a three-dimensional matrix. The first two dimensions represent the above-mentioned time grid, and the third dimension represents the feature sequence sampled at the corresponding moment, which is recorded as R represents the number of time grid rows, C represents the number of time grid columns, Represents the feature sequence in the rth row and cth column in the time grid.

[0056] (3) Initialize the designed multi-cycle semantic interaction network, which is a grid-like cyclic network. Figure 3 As shown, the designed semantic interaction units are arranged according to the above time grid, that is, the semantic interaction unit located in the Rth row and the Cth column is responsible for processing the feature sequence located in the Rth row and the Cth column in the three-dimensional matrix.

[0057] The semantic interaction unit is composed of two long short-term memory variants and a fusion gate. Each semantic interaction unit is responsible for processing the feature sequence sampled at the corresponding moment and transmitting the processed data to the adjacent semantic interaction units on its right and below, thereby utilizing the bidirectional recurrent structure to simultaneously mine multi-cycle correlations.

[0058] like Figure 4 As shown, the Multi-Period Semantic Interactive Network (MPSIN) has three output vectors, which are represented by H hor , H ver , Out means:

[0059] H hor ,H ver ,Out=MPSIN(X 2D ), (1)

[0060] in, is the horizontal loop output vector, is the vertical cycle output vector, L is the number of network layers, d model is the hidden unit dimension. is the output vector of the last layer.

[0061] The multi-cycle semantic interaction network is composed of the designed semantic interactive units (SIU) arranged according to the above time grid. The semantic interactive unit is composed of two long short-term memory variants and a fusion gate, such as Figure 5 As shown, the long short-term memory variant responsible for horizontal transmission is denoted as SIU hor , the long short-term memory variant responsible for longitudinal transmission is denoted as SIU ver , fusion gated o gate Responsible for fusing the horizontal transmission results with the vertical transmission results. Each semantic interaction unit is responsible for processing the feature sequence sampled at the corresponding moment. The specific process is as follows:

[0062]

[0063] Where r, c, and l in the subscripts represent the rth row (1≤r≤R), the cth column (1≤c≤C), and the lth layer (1≤l≤L) respectively. When l=1, Represents the input of the first layer. When l>1, That is, the output of the l-1th layer is used as the input of the lth layer. and They represent the horizontal input of the current unit (i.e. the output of the previous unit in the same row) and the vertical input (i.e. the output of the unit above in the same column). When r=1, Replace it with a vector of all 0s, when c = 1, Replace it with a vector of all 0s. is the output of the horizontal long-short time variant and is used as the input of the following unit in the same row; is the output of the vertical long-short time variant and is used as the input of the following unit in the same column; r,c,l is the output of the fused gate and is used as the input of the unit in the same row and column of the next layer.

[0064] Long short-term memory variants combine the advantages of LSTM and GRU, using both the gating concept and the gate reuse concept to reduce parameters. Figure 6 As shown in , LSTM has three gates: input gate, forget gate, and output gate. GRU has two gates: update gate and reset gate. Compared with LSTM and GRU, the semantic interaction unit has two gates: main control gate and auxiliary control gate, as shown in Figure 7 As shown, the long-term and short-term temporal correlations can be extracted separately during the cyclic transmission process. The working principle is as follows:

[0065]

[0066]

[0067] Among them, p t Represents the master control gate, s t Represents the auxiliary control door, represents the main hidden state, Represents a secondary hidden state. is the output at time t, is the main direction input at time t, is the secondary direction input at time t, x is the original input. σ and tanh represent the sigmoid and tanh activation functions respectively, e represents the Hardman product, is the weight parameter, is the bias parameter. Similarly, is the weight parameter, is the bias parameter.

[0068] For the long short-term memory variant SIU responsible for lateral transfer hor For example, as shown in formula (2), the inputs are x=input r,c,l , Its output As can be seen, lateral transmission corresponds to the connection direction of all time steps on the same day in the temporal network, with the output of the previous unit in the same row serving as the input for the primary direction, and the output of the previous unit in the same column serving as the input for the secondary direction. By distinguishing between primary and secondary directions, the network can be trained to automatically discover periodic patterns in the corresponding transmission direction based on learning from multiple aspects of information. Therefore, lateral circulation can fully exploit short-term temporal correlations.

[0069] For the long short-term memory variant SIU responsible for longitudinal transmission ver For example, as shown in formula (3), x = input r,c,l , Its output It can be seen that the vertical transmission corresponds to the connection direction of the same time point on different days in the time grid. The output of the upper unit in the same column is used as the input of the main direction, and the output of the previous unit in the same row is used as the input of the secondary direction. By distinguishing the main direction from the secondary direction, the network can be trained to automatically mine the periodic pattern in the corresponding transmission direction based on learning multiple aspects of information. Therefore, long-term time correlation can be fully mined through vertical circulation.

[0070] Fusion Gating gate , responsible for SIU hor and SIU ver The output of is fused to obtain multi-cycle time correlation, and the working process is as follows:

[0071]

[0072] o r,c,l =o t ef t , (12)

[0073] Among them, t represents the fusion gate, f t represents the fused hidden state, is the weight parameter, is the bias parameter. The meanings of the other parameters are the same as those described above.

[0074] Will Figure 4 The output result Out of the last layer is input to the prediction module composed of a multi-layer perceptron. The prediction module is responsible for mapping the intermediate results that integrate multiple time correlations into the prediction results.

[0075] (4) Use the preprocessed data samples to train the multi-period semantic interaction network to predict the feature values of future moments based on the feature values of past observations; divide the preprocessed data samples into a training set, a validation set, and a test set in a ratio of 6:2:2, which are used to train the prediction network, verify the training effect, and evaluate the prediction accuracy.

[0076] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A multivariate time series prediction method based on a multi-period semantic interaction network, characterized in that: include: S1, collects multivariate time series data at fixed time intervals; S2, for historical sequences, performs data preprocessing in the time dimension and feature dimension in the order from feature dimension to time dimension; S3, initialize the multi-cycle semantic interaction network; S4, using the data samples preprocessed by S1-S2, trains a multi-period semantic interaction network to predict the feature values of future moments based on the feature values of past observations; The preprocessed data samples are divided into training set, validation set and test set according to the proportion, which are used to train the prediction network, verify the training effect and evaluate the prediction accuracy.

2. The multivariate time series prediction method based on a multi-period semantic interaction network according to claim 1, characterized in that: In S1, the collected multivariate time series data is recorded as the original data set Where T represents the total number of sampled time steps, and N represents the number of features sampled at each time step. If the length of the historical sequence is H and the length of the predicted sequence is P, the historical sequence is recorded as The predicted sequence is recorded as The original data set is divided into data samples according to the length of the historical sequence and the length of the predicted sequence. The length of each data sample is H+P, and the historical sequence part and the predicted sequence part are marked separately. Among them, the historical sequence part is used as the network input, including the time dependency to predict the future sequence, and the predicted sequence part is used as the true value label to evaluate the prediction results.

3. The multivariate time series prediction method based on a multi-period semantic interaction network according to claim 1, characterized in that: In S2, In the feature dimension, the sampling sequence of the same feature is subjected to outlier detection and normalization; In the time dimension, the sampling time is re-divided into a two-dimensional grid structure according to the "sampling day" standard, that is, the time grid. Each row represents the sampling values of all time steps in the same day, and each column represents the sampling values of the same time point on different days; The preprocessed historical data is constructed into a three-dimensional matrix, where two dimensions represent the time grid and the other dimension represents the feature sequence sampled at the corresponding moment, which is recorded as R represents the number of time grid rows, C represents the number of time grid columns, Represents the feature sequence in the rth row and cth column in the time grid.

4. The multivariate time series prediction method based on a multi-period semantic interaction network according to claim 1, characterized in that: In S3, The multi-cycle semantic interaction network is a grid-like recurrent network, which is composed of semantic interaction units arranged according to the time grid. That is, the semantic interaction unit located in the Rth row and Cth column is responsible for processing the feature sequence located in the Rth row and Cth column in the three-dimensional matrix. The semantic interaction unit includes long short-term memory variants and fusion gates; Each semantic interaction unit processes the feature sequence sampled at the corresponding moment and transmits the processed data to the adjacent semantic interaction unit, using a bidirectional loop structure to simultaneously mine multi-cycle correlations.

5. The multivariate time series prediction method based on a multi-period semantic interaction network according to claim 4, characterized in that: In S3, the multi-cycle semantic interaction network has three output vectors, H hor , H ver , Out means: H hor ,H ver ,Out=MPSIN(X 2D ), (1) in, is the horizontal loop output vector, is the vertical cycle output vector, L is the number of network layers, d model is the hidden unit dimension, is the output vector of the last layer.

6. According to the multivariate time series prediction method based on a multi-period semantic interaction network of claim 4, in the semantic interaction unit, the long short-term memory variant includes a first long short-term memory variant and a second long short-term memory variant; wherein, The first long short-term memory variant responsible for horizontal transmission is denoted as SIU hor The second long short-term memory variant responsible for longitudinal transmission is denoted as SIU ver , fusion gated o gate Responsible for fusing the horizontal transmission results with the vertical transmission results; each semantic interaction unit is responsible for processing the feature sequence sampled at the corresponding moment. The specific process is as follows: Where r, c, and l in the subscripts represent the rth row (1≤r≤R), the cth column (1≤c≤C), and the lth layer (1≤l≤L), respectively; When l=1, Represents the input of the first layer. When l>1, That is, the output of the l-1 layer is used as the input of the l layer; and They represent the horizontal input of the current unit (i.e. the output of the previous unit in the same row) and the vertical input (i.e. the output of the unit above in the same column). When r=1, Replace it with a vector of all 0s, when c = 1, Replace with a vector of all 0s; is the output of the horizontal long-short time variant and is used as the input of the following unit in the same row; is the output of the vertical long-short time variant and is used as the input of the following unit in the same column; r,c,l is the output of the fused gate and is used as the input of the unit in the same row and column of the next layer.

7. According to the multivariate time series prediction method based on a multi-cycle semantic interaction network in claim 6, in the semantic interaction unit, the long short-term memory variant includes a main control gate and an auxiliary control gate, which respectively extract long-term and short-term temporal correlations during the cyclic transmission process. Its working principle is as follows: in, p t Represents the master control gate, s t Represents the auxiliary control door, represents the main hidden state, Represents a secondary hidden state. is the output at time t, is the main direction input at time t, is the secondary direction input at time t, and x is the original input; σ and tanh represent sigmoid and tanh activation functions respectively, e represents the Hardman product, is the weight parameter, is the bias parameter. Similarly, is the weight parameter, is the bias parameter; For the first long short-term memory variant SIU responsible for lateral transmission hor For example, as shown in formula (2), the inputs are x=input r,c,l , Its output Horizontal transmission corresponds to the connection direction of all time steps on the same day in the time network. The output of the previous unit in the same row is used as the input of the primary direction, and the output of the previous unit in the same column is used as the input of the secondary direction. By distinguishing between the primary and secondary directions, the network is trained based on learning multiple aspects of information to automatically discover the periodic patterns in the corresponding transmission direction, thereby fully exploring short-term temporal correlations through horizontal circulation. For the second long short-term memory variant SIU responsible for longitudinal transmission ver For example, as shown in formula (3), x = input r,c,l , Its output Vertical transmission corresponds to the connection direction of the same time point on different days in the time grid. The output of the upper unit in the same column is used as the input of the main direction, and the output of the previous unit in the same row is used as the input of the secondary direction. By distinguishing the main direction from the secondary direction, the network is trained on the basis of learning multiple aspects of information to automatically mine the periodic patterns in the corresponding transmission direction, so that long-term time correlation can be fully explored through vertical circulation.

8. According to the multivariate time series prediction method based on multi-cycle semantic interaction network of claim 6, the gated gate , responsible for SIU hor and SIU ver The output of is fused to obtain multi-cycle time correlation, and the working process is as follows: o r,c,l =o t ef t , (12) in, o t represents the fusion gate, f t represents the fused hidden state, is the weight parameter, is the bias parameter.

9. The multivariate time series prediction method based on a multi-period semantic interaction network according to claim 1, wherein in S4, The output vector of the last layer Connected to the prediction model composed of a multi-layer perceptron, the prediction module is responsible for mapping the intermediate results that integrate multiple time correlations into prediction results.

Images