A time series forecasting method and system based on diffusion autoregressive transformer

Abstract

This invention belongs to the field of time series prediction technology, specifically relating to a time series prediction method and system based on a diffusion autoregressive transformer. This method segments time series data into several segments and uses a self-attention mechanism to model the dependencies between segments. It also introduces diffusion and denoising mechanisms to effectively capture local features within segments. Furthermore, this invention, through the comprehensive application of segment embedding technology, self-attention mechanism, and cross-attention mechanism, not only effectively improves the model's ability to capture global dependencies in time series data but also significantly enhances the model's ability to finely model local features within segments. This innovative model architecture enables this invention to achieve higher accuracy and more stable performance in time series prediction tasks.

Description

Technical Field

[0001] This invention belongs to the field of time series forecasting technology, specifically relating to a time series forecasting method and system based on diffusion autoregressive transformer. Background Technology

[0002] In recent years, deep learning technology has made significant progress in the field of time series forecasting and has been widely applied in areas such as financial markets, energy management, and healthcare. These technologies can assist decision-makers in optimizing resource allocation, reducing risks, and driving the development of intelligent systems. Self-supervised learning (SSL), as one of the mainstream methods, significantly reduces its reliance on large amounts of labeled data by pre-training with learning signals automatically generated from unlabeled data. Among these, contrastive learning and masked autoencoders are two widely used SSL techniques. They train models by distinguishing between similar and dissimilar data and by masking parts of the input data, respectively, thereby learning effective feature representations and the latent structure of the data.

[0003] However, existing self-supervised learning methods still have limitations in capturing global dependencies and local features in time series data. Contrastive learning methods, when constructing positive and negative sample pairs, struggle to fully consider the temporal dependencies and ambiguity of the data, leading to poor model performance in extracting deep features. Masked autoencoders suffer from data distribution discrepancies during pre-training; the lack of masked data causes the model to become disconnected from real-world scenarios, affecting its generalization ability. Furthermore, existing methods often focus on only one aspect, lacking a unified framework capable of simultaneously modeling long-term and short-term dependencies in time series data. Summary of the Invention

[0004] The purpose of this invention is to provide a time series prediction method and system based on diffusion autoregressive transformers to solve the problems mentioned in the background art.

[0005] The present invention achieves the above objectives through the following technical solutions:

[0006] Firstly, this invention proposes a time series prediction method based on a diffusion autoregressive transformer, the method comprising:

[0007] Obtain the raw time series data of the object to be predicted and perform preprocessing;

[0008] The preprocessed time series data is divided into several subsequences;

[0009] The time-series dependencies of the subsequences are captured using an autoregressive transformer encoder module, and a diffusion process is introduced during the encoding process to generate a self-supervised signal.

[0010] Combining the self-supervised signal, the denoising autoregressive transformer decoder module utilizes the inverse diffusion process and cross-attention mechanism to fuse global and local features to generate a deep representation;

[0011] By transforming the dimension of the depth representation through a linear connection layer, an approximate representation of the original time series data is reconstructed as the prediction result.

[0012] Furthermore, the step of acquiring the original time series data of the object to be predicted and performing preprocessing includes:

[0013] Obtain the multivariate raw time series data X = [x1, x2, ..., x] of the object to be predicted. L ],X∈R C×L Where C represents the number of variables and L represents the backtracking window length;

[0014] Normalization is performed, and the normalization expression is: Where, x c,i This represents the i-th time series data in the c-th window. This represents the mean of all samples along this dimension. This represents the standard deviation of all samples in this dimension.

[0015] Furthermore, the step of dividing the preprocessed time series data into several subsequences includes:

[0016] The normalized time series data is divided into several segments using segment embedding technology, with each segment consisting of data points from adjacent time steps; after segmentation, the time series data x 1:L Convert to x 1:N , where L is the product of N and the segment length;

[0017] The length of each segment is determined based on a preset time window;

[0018] Each segment is mapped to a high-dimensional embedding space using embedding techniques to obtain the subsequence z. 1:N As shown in the following formula:

[0019] z 1:N =Embedding(x 1:N ).

[0020] Furthermore, the step of using an autoregressive transformer encoder module to capture the temporal dependencies of the subsequences and introducing a diffusion process to generate a self-supervised signal during the encoding process includes:

[0021] Construct an encoder module based on an autoregressive transformer architecture, using the subsequence As input, the temporal dependencies of each segment in the subsequence are captured through a multi-head attention mechanism, as shown in the following equation:

[0022]

[0023] Where SOS represents the start of sequence marker, PE represents position encoding, adding position encoding gives the sequence positional features to capture the temporal dependencies between segments; and f() represents the encoder.

[0024] A diffusion process is introduced into the encoded subsequence, progressively moving towards each time step. Add noise to the encoded features to generate a self-supervised signal. Where j = 1, ..., N, the noise is controlled by a cosine scheduler to vary its injection intensity and time step, as shown in the following equation:

[0025]

[0026] in α s =1-β s ,β s The noise figure with diffusion step size s is set as an increasing sequence and a cosine scheduling strategy is used.

[0027] Furthermore, the self-supervised signal is combined with a denoising autoregressive transformer decoder module that utilizes inverse diffusion and cross-attention mechanisms to fuse global and local features to generate a deep representation, including:

[0028] A decoder module based on a denoising autoregressive transformer architecture is constructed. The self-supervised signal is used as the target signal to guide the gradual removal of added noise through a reverse diffusion process. The global information from the encoding stage and the local features from the current decoding stage are fused using a cross-attention mechanism to generate a depth representation L1. The calculation formula is as follows:

[0029]

[0030] Where E represents the expectation, denoted by , represents the original distribution of the input time series, and ∈ represents the noise added at each step, which satisfies the standard Gaussian distribution N(0,1).

[0031] Furthermore, the step of transforming the dimensionality of the depth representation through a linear connection layer to reconstruct an approximate representation of the original time series data as a prediction result includes:

[0032] The depth representation is transformed in dimension through a linear connection layer to match the dimension of the original time series data. The calculation formula is as follows:

[0033]

[0034] Where f represents the output of the characterization network, and g represents the output of the decoder module and the linear connection layer;

[0035] An approximate representation of the original time series is reconstructed, and this approximate representation is output as the prediction result. The L2 calculation formula for the reconstructed approximate representation is as follows:

[0036]

[0037] Secondly, the present invention proposes a time series prediction system based on a diffusion autoregressive transformer, applied to perform the time series prediction method described in any of the above claims, the system comprising:

[0038] The data preprocessing module is used to acquire the raw time series data of the object to be predicted and perform preprocessing.

[0039] The data partitioning module is used to divide the preprocessed time series data into several subsequences;

[0040] The encoder module is used to capture the temporal dependencies of the subsequence using an autoregressive transformer encoder module, and to introduce a diffusion process to generate a self-supervised signal during the encoding process;

[0041] The decoder module is used to combine the self-supervised signal and, through the denoising autoregressive transformer decoder module, fuse global and local features using the inverse diffusion process and cross-attention mechanism to generate a deep representation;

[0042] The prediction output module is used to transform the dimension of the depth representation through a linear connection layer to reconstruct an approximate representation of the original time series data as the prediction result.

[0043] The beneficial effects of this invention are as follows:

[0044] This prediction method, by introducing diffusion and denoising mechanisms and combining them with an autoregressive transformer architecture, achieves unified modeling of global and local features in time series data. Traditional self-supervised learning methods often struggle to simultaneously capture global dependencies and local details in time series prediction. This invention, through the comprehensive application of fragment embedding techniques, self-attention mechanisms, and cross-attention mechanisms, not only effectively improves the model's ability to capture global dependencies in time series data but also significantly enhances its ability to finely model local features within fragments. This innovative model architecture enables this invention to achieve higher accuracy and more stable performance in time series prediction tasks. Attached Figure Description

[0045] Figure 1 A flowchart illustrating a time series prediction method based on a diffusion autoregressive transformer provided in an embodiment of this application;

[0046] Figure 2 A technical roadmap for a time series prediction method based on a diffusion autoregressive transformer provided in this application embodiment;

[0047] Figure 3 This is a schematic diagram of the structure of a time series prediction system based on a diffusion autoregressive transformer provided in an embodiment of this application. Detailed Implementation

[0048] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0049] Example 1

[0050] like Figure 1 and Figure 2 As shown in the figure, this embodiment proposes a time series prediction method based on diffusion autoregressive transformers, the method including the following:

[0051] (1) Data acquisition stage: Acquire multivariate time series data of the object to be predicted and perform normalization processing to obtain the original sequence data;

[0052] More specifically, obtain the multivariate raw time series data X = [x1, x2, ..., x...] of the object to be predicted. L ],X∈R C ×L Where C represents the number of variables and L represents the backtracking window length;

[0053] Normalization is performed, and the normalization expression is: Where, x c,i This represents the i-th time series data in the c-th window. This represents the mean of all samples along this dimension. This represents the standard deviation of all samples in this dimension.

[0054] (2) Data partitioning stage: The original sequence data is divided into several segments to obtain subsequence data, each of which consists of data points from adjacent time steps; after segmentation, the time series data x 1:L Can be converted to x 1:N , where L is the product of N and the segment length.

[0055] It should be noted here that in time series forecasting, the time step refers to the time interval between one observation point and the next observation point in the series data. It defines the sampling frequency and time scale of the time series data.

[0056] More specifically, the preprocessed time series data is divided into several subsequences, including:

[0057] The normalized time series data is divided into several segments using segment embedding technology, with each segment consisting of data points from adjacent time steps; after segmentation, the time series data x 1:L Can be converted to x 1:N , where L is the product of N and the segment length.

[0058] The length of each segment is determined based on a preset time window;

[0059] Each segment is mapped to a high-dimensional embedding space using embedding techniques to obtain the subsequence z. 1:N As shown in the following formula:

[0060] z 1:N =Embedding(x 1:N ).

[0061] (3) Model building stage: The encoder module based on the Autoregressive Transformer architecture and the decoder module based on the Denoising Autoregressive Transformer architecture take the subsequence data as input, capture the temporal dependencies of each segment in the subsequence data through the Multi-Head Attention mechanism, and build the TimeDART model.

[0062] (4) Encoding stage: In the encoder module, a diffusion process is introduced into the encoded subsequence data to gradually add noise to the encoded features at each time step and generate a self-supervised signal;

[0063] More specifically, the autoregressive transformer encoder module is used to capture the temporal dependencies of subsequences, and a diffusion process is introduced during the encoding process to generate self-supervised signals, including:

[0064] Construct an encoder module based on an autoregressive transformer architecture, using subsequences As input, the temporal dependencies of each segment in the subsequence are captured through a multi-head attention mechanism, as shown in the following equation:

[0065]

[0066] Where SOS represents the start of sequence marker, PE represents position encoding, adding position encoding gives the sequence positional features to capture the temporal dependencies between segments; f() represents the encoder, used for subsequent formula derivation.

[0067] A diffusion process is introduced into the encoded subsequence, gradually moving towards each time step. Add noise to the encoded features to generate a self-supervised signal. Where j = 1, ..., N, the noise is controlled by a cosine scheduler to vary its injection intensity and time step, as shown in the following equation:

[0068]

[0069] in α s =1-β s ,β s The noise figure with diffusion step size s is set as an increasing sequence and a cosine scheduling strategy is used.

[0070] (5) Decoding stage: In the decoder module, the self-supervised signal is used as the target signal, and the added noise is gradually removed through the reverse diffusion process. The global information of the encoding stage and the local features of the current decoding stage are fused using the cross attention mechanism to generate a depth representation.

[0071] By combining self-supervised signals, a denoising autoregressive transformer decoder module utilizes inverse diffusion and cross-attention mechanisms to fuse global and local features, generating a deep representation, including:

[0072] A decoder module based on a denoising autoregressive transformer architecture is constructed. The self-supervised signal is used as the target signal to guide the gradual removal of added noise through a reverse diffusion process. The cross-attention mechanism is used to fuse global information from the encoding stage with local features from the current decoding stage to generate a depth representation L1. The calculation formula is as follows:

[0073]

[0074] in α s =1-β s ,β s The noise figure with diffusion step size s is set as an increasing sequence and a cosine scheduling strategy is used.

[0075] In some optional embodiments, the derivation process of the above depth representation is as follows:

[0076] The decoder module is used to reconstruct the noisy signal during the reverse process. This module incorporates a cross-attention mechanism to adaptively map noisy segments to clean segments, effectively capturing local detail features and ensuring the model can recover high-quality time series segments. When using a neural network to denoise a noisy sequence, Bayes' theorem states that, given conditions... and When the conditional probability satisfies the following relationship:

[0077]

[0078] Expanding the Gaussian distribution, the above conditional probability can be written in the following form:

[0079]

[0080] A neural network is used to fit the above process, and the fitting result is p. θ (x s-1 │x s ), set as:

[0081]

[0082] The loss function can then be chosen as the KL divergence between the two:

[0083] D KL (q(x s-1 │x s ,x 0 )│p θ (x s-1 │x s ))

[0084] Substituting the KL divergence calculation, we can derive the final optimization objective as:

[0085]

[0086] (6) Prediction output stage: The depth representation is transformed through a linear connection layer to match the dimension of the original time series data, and an approximate representation of the original time series is reconstructed. The approximate representation is then output as the prediction result.

[0087] More specifically, by transforming the dimensionality of the depth representation through linear connection layers, an approximate representation of the original time series data is reconstructed as the prediction result, including:

[0088] The depth representation is transformed in dimension through a linear connection layer to match the dimension of the original time series data. The calculation formula is as follows:

[0089]

[0090] Where f represents the output of the characterization network, and g represents the output of the decoder module and the linear connection layer;

[0091] An approximate representation of the original time series is reconstructed, and this approximate representation is output as the prediction result. The L2 calculation formula for the reconstructed approximate representation is as follows:

[0092]

[0093] According to the above embodiments of the present invention, the TimeDART diffusion autoregressive transformer model combines the generative method of self-supervised learning with diffusion and autoregressive models to form a unified framework. The advantages of TimeDART in this embodiment are reflected in the following aspects:

[0094] 1. Unified modeling of global and local data: TimeDART effectively captures global dependencies in time series through an autoregressive transformer encoder, and further enhances the fine modeling of local features of segments by using diffusion and denoising mechanisms, thus enabling it to handle both long-term and short-term dependencies.

[0095] 2. Denoising Diffusion Mechanism: By introducing a diffusion process, the model can gradually incorporate noise during self-supervised learning, generating more diverse training samples and thus learning more robust feature representations. The introduction of the denoising decoder enables the model to adaptively recover from various noise levels, enhancing its predictive ability in uncertain data.

[0096] 3. Generative self-supervised learning: Compared with methods such as masked autoencoders, our diffusion autoregressive method narrows the gap between pre-training and fine-tuning tasks. The generative optimization approach is more in line with the natural generation process of time series data and can better capture the multimodal features of time series.

[0097] Combination Figure 3 Based on the same inventive concept, this embodiment also proposes a time series prediction system based on a diffusion autoregressive transformer, applied to the above-described time series prediction method. The system includes:

[0098] The data preprocessing module is used to acquire the raw time series data of the object to be predicted and perform preprocessing.

[0099] The data partitioning module is used to divide the preprocessed time series data into several subsequences;

[0100] The encoder module is used to capture the temporal dependencies of subsequences using an autoregressive transformer encoder module and to introduce a diffusion process to generate a self-monitoring signal during the encoding process;

[0101] The decoder module is used to combine the self-supervised signal and, through the denoising autoregressive transformer decoder module, fuse global and local features using the inverse diffusion process and cross-attention mechanism to generate a deep representation.

[0102] The prediction output module is used to transform the dimension of the depth representation through a linear connection layer, reconstructing an approximate representation of the original time series data as the prediction result.

[0103] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described time series prediction methods.

[0104] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the time series prediction methods described in the above embodiments.

[0105] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0106] In addition, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0107] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A time series forecasting method based on a diffusion autoregressive transformer, characterized in that, The method includes: Obtain the raw time series data of the object to be predicted and perform preprocessing; The preprocessed time series data is divided into several subsequences; The time-series dependencies of the subsequences are captured using an autoregressive transformer encoder module, and a diffusion process is introduced during the encoding process to generate a self-supervised signal. Combining the self-supervised signal, the denoising autoregressive transformer decoder module utilizes the inverse diffusion process and cross-attention mechanism to fuse global and local features to generate a deep representation; The depth representation is transformed by a linear connection layer to reconstruct an approximate representation of the original time series data, which is then used as the prediction result. The method of capturing the temporal dependencies of the subsequences using an autoregressive transformer encoder module and introducing a diffusion process to generate a self-supervised signal during the encoding process includes: Construct an encoder module based on an autoregressive transformer architecture, using the subsequence : ； SOS represents the start of sequence marker, PE represents position encoding. Adding position encoding gives the sequence positional features in order to capture the temporal dependencies between segments. This indicates an encoder; A diffusion process is introduced into the encoded subsequence, progressively moving towards each time step. Add noise to the encoded features to generate a self-supervised signal. ,in The noise is controlled by a cosine scheduler, which modulates the changes in its injection intensity and time step, as follows: ； in Indicates the diffusion step size is The noise figure is set as an increasing sequence and a cosine scheduling strategy is used. .

2. The time series prediction method based on diffusion autoregressive transformer according to claim 1, characterized in that: The process of acquiring and preprocessing the original time series data of the object to be predicted includes: Obtain the multivariate raw time series data of the object to be predicted. ,in Indicates the number of variables. Indicates the length of the backtracking window; Normalization is performed, and the normalization expression is: ;in, Indicates the first The first window One time series data, This represents the mean of all samples along this dimension. This represents the standard deviation of all samples in this dimension.

3. The time series prediction method based on diffusion autoregressive transformer according to claim 2, characterized in that: The step of dividing the preprocessed time series data into several subsequences includes: The normalized time series data is divided into several segments using segment embedding technology, with each segment consisting of data points from adjacent time steps; after segmentation, the time series data... Convert to , where L is the product of N and the segment length; The length of each segment is determined based on a preset time window; Each segment is mapped to a high-dimensional embedding space using embedding techniques to obtain subsequences. As shown in the following formula: 。 4. The time series prediction method based on diffusion autoregressive transformer according to claim 3, characterized in that: The self-supervised signal is combined with a denoising autoregressive transformer decoder module that uses inverse diffusion and cross-attention mechanisms to fuse global and local features to generate a deep representation, including: A decoder module based on a denoising autoregressive transformer architecture is constructed. The self-supervised signal is used as the target signal to guide the gradual removal of added noise through a reverse diffusion process. A cross-attention mechanism is used to fuse global information from the encoding stage with local features from the current decoding stage to generate a depth representation. The calculation formula is as follows: ； in, For the expectation, This represents the original distribution of the input time series. This indicates that the noise added at each step follows a standard Gaussian distribution. .

5. The time series prediction method based on diffusion autoregressive transformer according to claim 4, characterized in that: The step of transforming the dimensionality of the depth representation through a linear connection layer to reconstruct an approximate representation of the original time series data as a prediction result includes: The depth representation is transformed in dimension through a linear connection layer to match the dimension of the original time series data. The calculation formula is as follows: ； in This represents the output of the network. This represents the output of the decoder module and the linear connection layer; An approximate representation of the original time series is reconstructed, and this approximate representation is output as the prediction result. The calculation formula is as follows: 。 6. A time series prediction system based on a diffusion autoregressive transformer, characterized in that, The system is applied to performing the time series forecasting method according to any one of claims 1-5, the system comprising: The data preprocessing module is used to acquire the raw time series data of the object to be predicted and perform preprocessing. The data partitioning module is used to divide the preprocessed time series data into several subsequences; The encoder module is used to capture the temporal dependencies of the subsequence using an autoregressive transformer encoder module, and to introduce a diffusion process to generate a self-supervised signal during the encoding process; The decoder module is used to combine the self-supervised signal and, through the denoising autoregressive transformer decoder module, fuse global and local features using the inverse diffusion process and cross-attention mechanism to generate a deep representation; The prediction output module is used to transform the dimension of the depth representation through a linear connection layer to reconstruct an approximate representation of the original time series data as the prediction result.

Images