Time sequence prediction method for correlation between dynamically changing variables
By using interactive convolution modules and other layer structures to extract dynamic variable relationships in time series prediction models, the shortcomings of existing models in capturing dynamic changes and negative correlations are solved, improving prediction accuracy and reducing computational costs.
Patent Information
- Application Number
- CN202411057706.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-19
AI Technical Summary
Existing time series prediction models based on deep learning are difficult to capture variable relationships and negative correlations that change dynamically over time, and the self-attention mechanism leads to high computational costs, affecting prediction accuracy and resource utilization efficiency.
Using interactive convolution module, temporary delamination, linear projection layer, feedforward layer, fully connected layer and nonlinear activation function layer, dynamic variable relationships are extracted through independent large-size convolution kernels, and layer normalization and residuals are performed. 连接,增强泛化能力和特征学习,减少计算消耗。
Effectively capture the dynamic properties and negative correlations between variables, improve the accuracy of time series prediction, reduce calculation costs, and improve resource utilization efficiency.
Smart Images

Figure CN118982113A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of time series prediction, in particular to a time series prediction method for the mutual relationship between dynamically changing variables. Background Art
[0002] In today's digital society, time series prediction methods have made great progress and are widely used in various fields, such as weather, traffic flow, finance, energy consumption, etc. In recent years, with the development of deep learning, more and more researchers have used deep neural networks to improve time series prediction models. However, the existing time series prediction models based on deep learning still have the following defects:
[0003] 1. It is difficult to capture the dynamic relationship between variables over time:
[0004] Most previous models are based on the Transformer structure, which embeds the entire time series into variable tokens that the Transformer can process, and then uses a multi-head self-attention mechanism to capture the relationship between variables. However, this embedding method will cause the relationship between variables to be simplistic, resulting in insufficient capture of the relationship between variables, which will affect the accuracy of the prediction. However, in the real world, the relationship between variables changes dynamically over time. For example, at the current moment, there is a positive correlation, while at the next moment, the relationship between variables may be more correlated or negatively correlated.
[0005] Therefore, the Transformer-based structure is not well suited to capturing dynamically changing variable relationships in time series, which will affect the accuracy of prediction.
[0006] 2. Unable to capture the negative correlation between variables and high computational cost:
[0007] When the Transformer model previously used the self-attention mechanism to extract the relationship between variables, the Softmax layer would change the attention score from a negative number to a positive number, that is, the original negative correlation between variables would be changed into a very small positive correlation. This would lose the original negative correlation and instead learn the wrong positive correlation, which is not conducive to the extraction of the dependency relationship between variables and thus damages the prediction. In addition, the quadratic computational and time consumption brought by the attention mechanism is also unavoidable, which would result in a huge computational cost and is not conducive to the rational use of resources. Summary of the invention
[0008] In order to solve the above problems, the present invention provides a time series prediction method for the relationship between dynamically changing variables, including the following steps: S1. Establishing a time series prediction model and performing model training; S2. Inputting the time series into the time series prediction model; S3. Generating a future time series of preset length.
[0009] Furthermore, the time series prediction model specifically includes: a padding layer, a residual layer, an interactive convolution module, a temporary retreat layer, a linear projection layer, a feedforward layer, a fully connected layer, and a nonlinear activation function layer.
[0010] Furthermore, the step S1 specifically includes the following sub-steps:
[0011] S1-1. Construct a training task data set and preprocess the training task data set;
[0012] S1-2. Input the processed data into the interactive convolution module and extract the variable relationship that changes dynamically over time;
[0013] S1-3. After the variable relationship is extracted, the output data passes through a temporary retreat layer and adds a residual connection to enhance the generalization ability;
[0014] S1-4. After transposing the time dimension and the variable dimension, the univariate time series is projected into a variable representation through a linear projection layer;
[0015] S1-5. Normalize the variable representation layer and input it into the feedforward layer network for further learning;
[0016] S1-6. Input to the fully connected layer for further enhanced feature learning;
[0017] S1-7. After strengthening the nonlinear ability through the nonlinear activation function layer, enter the temporary retreat layer and strengthen the generalization ability;
[0018] S1-8. Enter the fully connected layer again to make the hidden dimension consistent with the variable dimension after embedding;
[0019] S1-9. Obtain the variable representation of the future time series by regressing the layer and the residual layer and performing layer normalization again;
[0020] S1-10. Linearly project the variable representation of the future time series into the future time series.
[0021] Furthermore, the training task data set in the S1-1 step is specifically the historical data of the prediction task, and the data set is divided into a training set, a validation set, and a test set according to a ratio; the division ratio of the data set is 7:2:1 or 6:2:2.
[0022] Furthermore, the step S1-2 specifically includes the following sub-steps:
[0023] S1-2-1. For input data X∈R LxN Fill in variables at each time step;
[0024] S1-2-2. Fill the last N-1 variables of the input data X to the front: in represents the jth time step;
[0025] S1-2-3. Apply independent large-size convolution kernels to the padded data at each time step to extract the variable relationship that changes dynamically over time;
[0026] The calculation formula for using an independent large-size convolution kernel to extract the variable relationship that changes dynamically over time is specifically: Among them, c j ∈R lxN represents the output after the variable relationship is extracted at the jth time step, w j =[w 1 j , w 2 j , ..., w N j ] represents the convolution kernel parameters at the jth time step.
[0027] Furthermore, the specific calculation formula for layer normalization in step S1-5 is: Among them, h n Represents the input data that needs to be layer normalized at the nth layer in the deep network.
[0028] Furthermore, the method further includes a time series refinement step: the prediction is refined by minimizing the mean square error formula; wherein the formula for minimizing the mean square error is specifically: In the formula, x ~ L+l represents the predicted future time point, x L+i Represents the true value.
[0029] The present invention provides a time series prediction method for the relationship between dynamically changing variables, which has the following beneficial effects:
[0030] The present invention solves the time series prediction task based on deep learning, the problem that it is difficult to capture the dynamic properties and negative correlation of the relationship between variables when using a model based on the Transformer architecture to extract the dependency between variables in the time series, and the cost-related problems of huge computational consumption and time consumption brought by the self-attention mechanism. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0032] Figure 1 A flow chart of the method provided by the present invention;
[0033] Figure 2 A schematic diagram of the structure of the interactive convolution module provided by the present invention. DETAILED DESCRIPTION
[0034] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0035] The following is a detailed description of the implementation method of the present invention in conjunction with the accompanying drawings. Only some embodiments are described, not all embodiments. For the purpose of clarity, representations and descriptions that are not related to the present invention are omitted in the drawings and descriptions.
[0036] In order to have a clearer understanding of the technical features, purposes and beneficial effects of the present invention, the technical solution of the present invention is now described in detail below. Obviously, the implementation cases described are part of the embodiments of the present invention, not all of the embodiments, and cannot be understood as limiting the scope of the implementation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.
[0037] like Figure 1 As shown, the present invention provides a time series prediction method for the relationship between dynamically changing variables, including the following steps: S1. Establishing a time series prediction model and performing model training; S2. Inputting the time series into the time series prediction model; S3. Generating a future time series of preset length.
[0038] Among them, the time series prediction model specifically includes: padding layer, residual layer, interactive convolution module (such as Figure 2 As shown), temporary retreat layer, linear projection layer, feedforward layer, fully connected layer, and nonlinear activation function layer.
[0039] Step S1 specifically includes the following sub-steps:
[0040] S1-1. Construct a training task dataset and preprocess the training task dataset.
[0041] Among them, the training task data set in step S1-1 is specifically the historical data of the prediction task, and the data set is divided into a training set, a validation set, and a test set according to a ratio; the division ratio of the data set is 7:2:1 or 6:2:2; and the data is divided into a fixed window size, and further divided into a fixed input length and output length in the window to suit model parallel training.
[0042] Preprocess the training task dataset: Normalization is a common technique to calibrate the distribution of input data. In time series forecasting, local statistics of history are usually removed to stabilize the forecast and restore these statistics to the model forecast. Following the common practice in many state-of-the-art models, reversible instance normalization is applied to the input sequence to center the sequence to zero mean, scale them to unit variance, and reverse normalize the forecast sequence; this preprocessing operation is defined as a job with hyperparameters, indicating whether to implement reversible instance normalization on the input data.
[0043] S1-2. Input the processed data into the interactive convolution module and extract the variable relationship that changes dynamically over time. Based on the time step independence strategy, independent convolution operations are used at each time step to capture the dependency relationship between variables. This ensures that the variable relationship learned at each time step is different, thereby extracting the dynamically changing variable relationship to improve the prediction accuracy. The specific steps include the following:
[0044] First, input data X∈R LxN Fill in the variables at each time step. Let X = [x1, x2, ..., x L ] where for the jth time step x j ∈Rl xN And x j =[x 1 j , x 2 j , ..., x N j ], fill the last N-1 variables to the front to become in Then, an independent large-size convolution kernel is applied to the padded data at each time step to extract the variable relationship that changes dynamically over time. The size of the convolution kernel is equal to the number of variables, which is N. When the convolution kernel with a kernel size of N slides along the variable dimension on the padded data, each slide can interact with all N variables, and then each time the relationship between all variables is extracted and the corresponding output is generated.
[0045] Among them, the calculation formula for using independent large-size convolution kernels to extract the variable relationship that changes dynamically over time is as follows: Among them, c j ∈RlxN represents the output after the variable relationship is extracted at the jth time step, w j =[w 1 j , w 2 j , ..., w N j ] represents the convolution kernel parameters at the jth time step.
[0046] In this way, the variable dependency relationship within each time step is fully extracted, and because the convolution kernel parameters are positive and negative, this also ensures the acquisition of negatively correlated variable relationships and the acquisition of different variable relationships at different time points.
[0047] S1-3. After the variable relationship is extracted, the output data passes through a temporary retreat layer and adds a residual connection to prevent overfitting and enhance generalization ability.
[0048] S1-4. Output C∈R after fully extracting the dynamic variable relationship in the previous step LxN , after transposing the time dimension and the variable dimension, the univariate time series is projected into a variable representation through a linear projection layer. That is: X emb =Embadding(C); where X emb ∈R NxD , D represents the feature dimension of the variable. All variables share the linear layer parameters, and all input window lengths are projected into the feature dimension of the variable.
[0049] S1-5. Normalize the variable representation layer by layer and then input it into the feedforward layer network for further learning: First, the variable representation will pass through the layer normalization layer. Normalizing the variable representation layer by layer helps to reduce the differences caused by inconsistent measurements in the original input data, and can improve the convergence of the deep network and the stability of training. The specific calculation formula for layer normalization is: Among them, h n Represents the input data that needs to be layer normalized at the nth layer in the deep network.
[0050] S1-6. Input into the fully connected layer to further strengthen feature learning; increasing or decreasing the feature dimension through the fully connected layer helps to learn richer variable features or extract more critical features, which helps to deepen the learning of variable representation and improve prediction accuracy.
[0051] S1-7. After strengthening the nonlinear ability through the nonlinear activation function layer, such as Gelu, Relu and other activation functions, the nonlinear ability is strengthened, the learning of variable representation is enriched, and the temporary retreat layer is entered to strengthen the generalization ability.
[0052] S1-8. Enter the fully connected layer again to make the hidden dimension consistent with the variable dimension after embedding.
[0053] S1-9. Through the temporary retreat layer and the residual layer and re-layer normalization, the variable representation of the future time series is obtained. The format of the linear layer is consistent with the linear layer format of the embedding layer.
[0054] S1-10. Linearly project the variable representation of the future time series into the future time series.
[0055] It also includes a time series refinement step: the forecast is refined by minimizing the mean square error formula; the formula for minimizing the mean square error is specifically: In the formula, x ~ L+l represents the predicted future time point, x L+i Represents the true value.
[0056] A time series prediction method that extracts variable relationships that change dynamically over time. The purpose is to extract more diverse variable relationships by capturing variable relationships that change dynamically over time to improve prediction accuracy. Therefore, it is applicable to the field of multivariate time series prediction.
[0057] Example 1: On a public weather time series dataset, the data is collected by the Max Planck Institute for Biogeochemistry. The dataset consists of 21 meteorological indicator data collected every ten minutes in 2020, including different meteorological indicators such as air humidity and temperature, which are also used as variables in the multivariate time series.
[0058] This embodiment is trained and predicted on the meteorological data set. Given an input time series with a backoff window of 960 minutes, that is, an input sequence with a length of 96, the predicted time series is also an output sequence with a length of 96, and the number of variables is 21. The specific implementation method is:
[0059] S1. Determine the training batch size and use only a single-core GPU for training. Set the backoff window and prediction window lengths to 96, the number of variables to 21, and set the training set: validation set: test set = 7:1:2.
[0060] S2, input the data into the model to train and update the model parameters, set the appropriate learning rate and training rounds until the results converge;
[0061] S3. The converged model can be used to predict future time series.
[0062] The above process is the general process of using the model to complete the prediction of future time series on the weather dataset.
[0063] Embodiment 2: A method for applying the model to traffic flow prediction is provided, the purpose of which is to extract the relationship between each traffic sensor node and make the flow value of each sensor in the future period of time, including the following steps:
[0064] A1. Collect data sets, i.e., the traffic monitoring values of sensors at each traffic intersection. The sampling frequency can be set to 1 hour, and the values of each sensor every hour are used as the multivariate values at a time point. For example, the hourly road occupancy rate measured by 862 sensors on the highways in the San Francisco Bay Area in the public data set is collected from January 2015 to December 2016;
[0065] A2. Set the training set, validation set and test set as in Example 1. This example also selects the historical backoff window length as 96, predicts the predicted value of each sensor for the next 96 hours, and sets the number of variables to 862;
[0066] A3. Train the model until convergence, and use the converged parameters for prediction. If you need to predict a longer value, you need to retrain the model and set the prediction length to the required value;
[0067] The above process is the specific application of this model in traffic flow prediction.
[0068] Embodiment 3: A method for extracting dynamically changing variable dependencies using an interactive convolution module is provided, the purpose of which is to extract diverse variable relationships that dynamically change over time, comprising the following steps:
[0069] A1. Multivariate time series X∈R to extract variable relationships L×N , L represents the sequence length, N represents the number of variables, and the sequence is input into the interactive convolution module;
[0070] A2. The interactive convolution module first performs variable filling operation on the multivariate time series X, and turns it into
[0071] A3. Then, at each time point, an independent convolution kernel with a kernel size of N is used to extract the variable relationship. Each time the convolution kernel slides along the variable dimension, the relationship between all variables can be captured.
[0072] A4. After capturing the variable relationship, you can choose whether to use a temporary retreat layer and a residual layer to prevent overfitting and enhance the generalization ability of the model;
[0073] The above process is the process of using the interactive convolution module to extract the variable relationship of any dynamically changing multivariate time series. This module can be used as a plug-and-play module to extract variable relationships on any model.
[0074] The present invention solves the time series prediction task based on deep learning, the problem that it is difficult to capture the dynamic properties and negative correlation of the relationship between variables when using a model based on the Transformer architecture to extract the dependency between variables in the time series, and the cost-related problems of huge computational consumption and time consumption brought by the self-attention mechanism.
[0075] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.
Claims
1. A time series prediction method for the relationship between dynamically changing variables, characterized in that: The following steps are involved: S1. Establish a time series prediction model and perform model training; S2. Input the time series into the time series prediction model; S3. Generate a future time series of preset length.
2. The time series prediction method for the dynamic relationship between variables according to claim 1 is characterized in that: The time series prediction model specifically includes: a padding layer, a residual layer, an interactive convolution module, a temporary retreat layer, a linear projection layer, a feedforward layer, a fully connected layer, and a nonlinear activation function layer.
3. The time series prediction method for the dynamic relationship between variables according to claim 2 is characterized in that: The S1 step specifically includes the following sub-steps: S1-1. Construct a training task data set and preprocess the training task data set; S1-2. Input the processed data into the interactive convolution module and extract the variable relationship that changes dynamically over time; S1-3. After the variable relationship is extracted, the output data passes through a temporary retreat layer and adds a residual connection to enhance the generalization ability; S1-4. After transposing the time dimension and the variable dimension, the univariate time series is projected into a variable representation through a linear projection layer; S1-5. After layer normalization, the variable representation is input into the feedforward layer network for further learning; S1-6. Input to the fully connected layer for further enhanced feature learning; S1-7. After strengthening the nonlinear ability through the nonlinear activation function layer, enter the temporary retreat layer and strengthen the generalization ability; S1-8. Enter the fully connected layer again to make the hidden dimension consistent with the variable dimension after embedding; S1-9. Obtain the variable representation of the future time series by regressing the layer and the residual layer and performing layer normalization again; S1-10. Linearly project the variable representation of the future time series into the future time series.
4. The time series prediction method for the dynamic relationship between variables according to claim 3 is characterized in that: The training task data set in the step S1-1 is specifically the historical data of the prediction task, and the data set is divided into a training set, a validation set, and a test set according to a ratio; the division ratio of the data set is 7:2:1 or 6:2:
2.
5. The time series prediction method for the dynamic relationship between variables according to claim 3 is characterized in that: The S1-2 step specifically includes the following sub-steps: S1-2-1. For input data X∈R LxN Fill in variables at each time step; S1-2-2. Fill the last N-1 variables of the input data X to the front: in represents the jth time step; S1-2-3. Apply independent large-size convolution kernels to the padded data at each time step to extract the variable relationship that changes dynamically over time; The calculation formula for using an independent large-size convolution kernel to extract the variable relationship that changes dynamically over time is specifically: Among them, c j ∈R lxN represents the output after the variable relationship is extracted at the jth time step, w j =[w 1 j , w 2 j , ..., w N j ] represents the convolution kernel parameters at the jth time step.
6. The time series prediction method for the dynamic relationship between variables according to claim 3 is characterized in that: The specific calculation formula for layer normalization in the S1-5 step is: Among them, h n Represents the input data that needs to be layer normalized at the nth layer in the deep network.
7. The time series prediction method for the dynamic relationship between variables according to claim 1 is characterized in that: It also includes a time series refinement step: the forecast is refined by minimizing the mean square error formula; the formula for minimizing the mean square error is specifically: In the formula, x ~ L+l represents the predicted future time point, x L+i Represents the true value.
