Time sequence prediction method and system based on diffusion autoregression transformer

Abstract

The invention belongs to the technical field of time series prediction, and particularly relates to a time series prediction method and system based on a diffusion autoregression transformer, and the method comprises the steps: segmenting time series data into a plurality of segments, and carrying out the modeling of a dependency relationship between the segments through employing a self-attention mechanism. And a diffusion and denoising mechanism is introduced, so that local features in the fragment are effectively captured. According to the method, through comprehensive application of a fragment embedding technology, a self-attention mechanism and a cross attention mechanism, the capture capability of the model on the global dependency of the time sequence is effectively improved, and the fine modeling capability of the model on the local features in the fragment is remarkably enhanced. By means of the innovative model architecture, higher precision and more stable performance can be obtained in a time sequence prediction task.

Description

Technical Field

[0001] The present invention belongs to the technical field of time series prediction, and in particular relates to a time series prediction method and system based on a diffuse autoregressive transformer. Background Art

[0002] In recent years, deep learning technology has made significant progress in the field of time series forecasting and has been widely used in financial markets, energy management, and healthcare. These technologies can help decision makers optimize resource allocation, reduce risks, and promote the development of intelligent systems. Self-supervised learning (SSL), as one of the mainstream methods, greatly reduces the dependence on a large amount of labeled data by automatically generating learning signals from unlabeled data for pre-training. Among them, contrastive learning and masked autoencoders are two widely used self-supervised learning techniques, which train models by distinguishing similar and dissimilar data and masking part of the input data, respectively, to learn effective feature representations and the potential structure of the data.

[0003] However, existing self-supervised learning methods still have limitations in capturing global dependencies and local features in time series data. Contrastive learning methods have difficulty in fully considering the temporal dependencies of time series and the ambiguity of data when constructing positive and negative sample pairs, resulting in poor performance of the model when extracting deep features. Masked autoencoders have data distribution differences during pre-training, and the lack of masked data makes the model out of touch with real scenarios in practical applications, affecting the generalization ability of the model. In addition, existing methods often only focus on a single aspect and lack a unified framework that can simultaneously model long-term and short-term dependencies in time series. Summary of the invention

[0004] The purpose of the present invention is to provide a time series prediction method and system based on a diffuse autoregressive transformer to solve the problems raised in the background technology.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] In a first aspect, the present invention proposes a time series prediction method based on a diffuse autoregressive transformer, the method comprising:

[0007] Obtain the original time series data of the object to be predicted and perform preprocessing;

[0008] Split the preprocessed time series data into several subsequences;

[0009] An autoregressive transformer encoder module is used to capture the temporal dependencies of the subsequences, and a diffusion process is introduced in the encoding process to generate a self-supervisory signal;

[0010] Combined with the self-supervisory signal, a denoising autoregressive transformer decoder module is used to fuse global and local features using an inverse diffusion process and a cross-attention mechanism to generate a deep representation;

[0011] The depth representation is transformed in dimension through a linear connection layer to reconstruct an approximate representation of the original time series data as a prediction result.

[0012] Furthermore, the obtaining of the original time series data of the object to be predicted and preprocessing includes:

[0013] Get the multivariate original time series data X = [x1, x2, ..., x L ],X∈R C×L , where C represents the number of variables and L represents the length of the lookback window;

[0014] After normalization, the normalized expression is: Among them, x c,i represents the i-th time series data on the c-th window, represents the mean of all samples on this dimension, Represents the standard deviation of all samples on this dimension.

[0015] Furthermore, the preprocessed time series data is divided into a plurality of subsequences, including:

[0016] The normalized time series data is divided into several segments using the fragment embedding technology. Each segment consists of data points at adjacent time steps. After segmentation, the time series data x 1:L Convert to x 1:N , where L is the product of N and the fragment length;

[0017] Determine the length of each segment based on a preset time window;

[0018] Through embedding technology, each fragment is mapped to a high-dimensional embedding space to obtain a subsequence z 1:N , as follows:

[0019] z 1:N =Embedding(x 1:N ).

[0020] Furthermore, the use of the autoregressive transformer encoder module to capture the temporal dependencies of the subsequences and introducing a diffusion process in the encoding process to generate a self-supervisory signal includes:

[0021] Construct an encoder module based on the autoregressive transformer architecture to As input, the temporal dependencies of each fragment in the subsequence are captured through a multi-head attention mechanism, as follows:

[0022]

[0023] Where SOS represents the sequence start marker, PE represents the position encoding, and adding the position encoding makes the sequence have position features to capture the temporal dependencies between each segment; f() represents the encoder;

[0024] Introducing a diffusion process into the encoded subsequence, gradually spreading to each time step Add noise to the encoded features to generate a self-supervisory signal Where j = 1, ..., N, the noise is controlled by the cosine scheduler to change its injection intensity and time step, as shown in the following formula:

[0025]

[0026] in α s =1-β s , β s Represents the noise coefficient with a diffusion step size of s, set it as an increasing sequence, and use the cosine scheduling strategy

[0027] Furthermore, the self-supervisory signal is combined with the denoising autoregressive transformer decoder module to fuse global and local features using an inverse diffusion process and a cross attention mechanism to generate a deep representation, including:

[0028] Construct a decoder module based on the denoising autoregressive transformer architecture, take the self-supervisory signal as the target signal, guide the gradual removal of the added noise through the inverse diffusion process, and use the cross-attention mechanism to fuse the global information of the encoding stage with the local features of the current decoding stage to generate the deep representation L1. The calculation formula is as follows:

[0029]

[0030] Where E is the expectation, represents the original distribution of the input time series, ∈ represents the noise added at each step, which satisfies the standard Gaussian distribution N(0,1).

[0031] Furthermore, the depth representation is transformed into a dimension through a linear connection layer to reconstruct an approximate representation of the original time series data as a prediction result, including:

[0032] The depth representation is transformed through a linear connection layer to match the dimension of the original time series data. The calculation formula is as follows:

[0033]

[0034] Where f represents the output of the representation network, and g represents the output of the decoder module and the linear connection layer;

[0035] Reconstruct an approximate representation of the original time series, and output the approximate representation as the prediction result. The L2 calculation formula of the reconstructed approximate representation is as follows:

[0036]

[0037] In a second aspect, the present invention proposes a time series prediction system based on a diffuse autoregressive transformer, which is applied to execute any of the time series prediction methods described above, and the system comprises:

[0038] The data preprocessing module is used to obtain the original time series data of the object to be predicted and perform preprocessing;

[0039] Data partitioning module, used to split the preprocessed time series data into several subsequences;

[0040] An encoder module, used for capturing the temporal dependency of the subsequences using an autoregressive transformer encoder module, and introducing a diffusion process in the encoding process to generate a self-supervisory signal;

[0041] A decoder module for combining the self-supervisory signal to generate a deep representation by fusing global and local features using an inverse diffusion process and a cross-attention mechanism through a denoising autoregressive transformer decoder module;

[0042] The prediction output module is used to transform the dimension of the depth representation through a linear connection layer and reconstruct an approximate representation of the original time series data as a prediction result.

[0043] The beneficial effects of the present invention are:

[0044] This prediction method achieves unified modeling of global and local features in time series data by introducing diffusion and denoising mechanisms and combining them with the autoregressive transformer architecture. Traditional self-supervised learning methods often find it difficult to capture global dependencies and local detail features simultaneously in time series prediction. The present invention not only effectively improves the model's ability to capture the global dependencies of time series through the comprehensive application of fragment embedding technology, self-attention mechanism, and cross-attention mechanism, but also significantly enhances the model's ability to finely model local features within fragments. This innovative model architecture enables the present invention to achieve higher accuracy and more stable performance in time series prediction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a flow chart of a time series prediction method based on a diffuse autoregressive transformer provided in an embodiment of the present application;

[0046] Figure 2 A technical roadmap for a time series prediction method based on a diffuse autoregressive transformer provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of the structure of a time series prediction system based on a diffuse autoregressive transformer provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The present application is further described in detail below in conjunction with the accompanying drawings. It is necessary to point out here that the following specific implementation methods are only used to further illustrate the present application and cannot be understood as limiting the scope of protection of the present application. Technical personnel in this field can make some non-essential improvements and adjustments to the present application based on the above application content.

[0049] Example 1

[0050] like Figure 1 and Figure 2 As shown, this embodiment proposes a time series prediction method based on a diffuse autoregressive transformer, and the method includes the following steps:

[0051] (1) Data acquisition stage: Obtain the multivariate time series data of the object to be predicted and perform normalization processing to obtain the original series data;

[0052] More specifically, obtain the multivariate original time series data X = [x1, x2, ..., x L ],X∈R C ×L , where C represents the number of variables and L represents the length of the lookback window;

[0053] After normalization, the normalized expression is: Among them, x c,i represents the i-th time series data on the c-th window, represents the mean of all samples on this dimension, Represents the standard deviation of all samples on this dimension.

[0054] (2) Data division stage: The original sequence data is divided into several segments to obtain subsequence data, each of which is composed of data points in adjacent time steps; after segmentation, the time series data x 1:L can be converted to x 1:N , where L is the product of N and the fragment length.

[0055] It should be noted here that in time series forecasting, the time step refers to the time interval between one observation point to the next observation point in the sequence data, which defines the sampling frequency and time scale of the time series data.

[0056] More specifically, the preprocessed time series data is divided into several subsequences, including:

[0057] The normalized time series data is divided into several segments using the fragment embedding technology. Each segment consists of data points at adjacent time steps. After segmentation, the time series data x 1:L can be converted to x 1:N , where L is the product of N and the fragment length.

[0058] Determine the length of each segment based on a preset time window;

[0059] Through embedding technology, each fragment is mapped to a high-dimensional embedding space to obtain a subsequence z 1:N , as follows:

[0060] z 1:N =Embedding(x 1:N ).

[0061] (3) Model construction stage: an encoder module based on an autoregressive transformer architecture and a decoder module based on a denoising autoregressive transformer architecture take the subsequence data as input, capture the temporal dependency of each segment in the subsequence data through a multi-head attention mechanism, and construct a diffusion autoregressive transformer model (TimeDART) model;

[0062] (4) Encoding stage: In the encoder module, a diffusion process is introduced into the encoded subsequence data to gradually add noise to the encoded features of each time step to generate a self-supervisory signal;

[0063] More specifically, the autoregressive transformer encoder module is used to capture the temporal dependencies of subsequences, and a diffusion process is introduced in the encoding process to generate self-supervisory signals, including:

[0064] Construct an encoder module based on the autoregressive transformer architecture to train the subsequence As input, the temporal dependencies of each fragment in the subsequence are captured through a multi-head attention mechanism, as follows:

[0065]

[0066] SOS represents the sequence start marker, PE represents the position encoding, and adding the position encoding makes the sequence have position features to capture the temporal dependencies between each fragment; f() represents the encoder, which is used for the subsequent formula derivation.

[0067] Introducing a diffusion process into the encoded subsequence, gradually spreading to each time step Add noise to the encoded features to generate a self-supervisory signal Where j = 1, ..., N, the noise is injected by the cosine scheduler to control the change of its intensity and time step, as shown in the following formula:

[0068]

[0069] in α s =1-β s , β s Represents the noise coefficient with a diffusion step size of s, set it as an increasing sequence, and use the cosine scheduling strategy

[0070] (5) Decoding stage: In the decoder module, the self-supervisory signal is used as the target signal, and the added noise is gradually removed through the inverse diffusion process. The cross-attention mechanism is used to fuse the global information of the encoding stage with the local features of the current decoding stage to generate a deep representation.

[0071] Combined with the self-supervisory signal, the denoising autoregressive transformer decoder module uses the inverse diffusion process and the cross-attention mechanism to fuse global and local features and generate deep representations, including:

[0072] Construct a decoder module based on the denoising autoregressive transformer architecture, take the self-supervisory signal as the target signal, guide the gradual removal of added noise through the inverse diffusion process, and use the cross-attention mechanism to fuse the global information in the encoding stage with the local features in the current decoding stage to generate the deep representation L1. The calculation formula is as follows:

[0073]

[0074] in α s =1-β s , β s Represents the noise coefficient with a diffusion step size of s, set it as an increasing sequence, and use the cosine scheduling strategy

[0075] In some optional embodiments, the derivation process of the above depth representation is as follows:

[0076] The decoder module is used to restore the noise signal in the reverse process. The decoder module combines the cross-attention mechanism to adaptively map the noise segment with the clean segment, thereby effectively capturing the local detail features of the segment and ensuring that the model can restore high-quality time series segments. When using a neural network to denoise a noisy sequence, the Bayesian formula shows that under given conditions and When , the conditional probability satisfies the following relationship:

[0077]

[0078] By expanding the Gaussian distribution, the above conditional probability can be written as follows:

[0079]

[0080] A neural network is used to fit the above process, and the fitting result is p θ (x s-1 │x s ), set it as:

[0081]

[0082] Then the loss function can be chosen as the KL divergence of the two:

[0083] D KL (q(x s-1 │x s ,x 0 )│p θ (x s-1 │x s ))

[0084] By bringing in the calculation of KL divergence, we can conclude that the final optimization goal is:

[0085]

[0086] (6) Prediction output stage: The deep representation is transformed through a linear connection layer to match the dimension of the original time series data, an approximate representation of the original time series is reconstructed, and the approximate representation is output as a prediction result.

[0087] More specifically, the depth representation is transformed into a dimension through a linear connection layer to reconstruct an approximate representation of the original time series data as a prediction result, including:

[0088] The depth representation is transformed through a linear connection layer to match the dimension of the original time series data. The calculation formula is as follows:

[0089]

[0090] Where f represents the output of the representation network, and g represents the output of the decoder module and the linear connection layer;

[0091] Reconstruct an approximate representation of the original time series and output the approximate representation as the prediction result. The L2 calculation formula of the reconstructed approximate representation is as follows:

[0092]

[0093] According to the above embodiment of the present invention, the above diffusion autoregressive transformer model (TimeDART) combines the generative method of self-supervised learning with the diffusion model and the autoregressive model to form a unified framework. The advantages of TimeDART in this embodiment are reflected in the following aspects:

[0094] 1. Unified modeling of global and local: TimeDART effectively captures the global dependencies in time series through the autoregressive transformer encoder, and further strengthens the fine modeling of local features of fragments by using diffusion and denoising mechanisms, so as to deal with long-term and short-term dependency problems at the same time.

[0095] 2. Denoising diffusion mechanism: By introducing the diffusion process, the model can gradually add noise in self-supervised learning, generate more diverse training samples, and then learn more robust feature representations. The introduction of the denoising decoder enables the model to perform adaptive recovery under multiple noise levels, enhancing the model's predictive ability in uncertain data.

[0096] 3. Generative self-supervised learning: Compared with methods such as masked autoencoders, our diffusion autoregression method narrows the gap between pre-training and fine-tuning tasks. The generative optimization method is more in line with the natural generation process of time series data and can better capture the multimodal characteristics of time series.

[0097] Combination Figure 3 Based on the same inventive concept, this embodiment also proposes a time series prediction system based on a diffuse autoregressive transformer, which is applied to execute the above-mentioned time series prediction method. The system includes:

[0098] The data preprocessing module is used to obtain the original time series data of the object to be predicted and perform preprocessing;

[0099] Data partitioning module, used to split the preprocessed time series data into several subsequences;

[0100] The encoder module is used to capture the temporal dependencies of subsequences using the autoregressive transformer encoder module and introduce a diffusion process in the encoding process to generate a self-supervisory signal;

[0101] The decoder module is used to combine the self-supervisory signal and generate deep representations by fusing global and local features using the inverse diffusion process and cross-attention mechanism through the denoising autoregressive transformer decoder module;

[0102] The prediction output module is used to transform the dimension of the deep representation through a linear connection layer and reconstruct an approximate representation of the original time series data as the prediction result.

[0103] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned time series prediction methods are implemented.

[0104] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the time series prediction relationship methods in the above embodiments.

[0105] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0106] In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0107] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A time series prediction method based on diffuse autoregressive transformer, characterized in that: The method comprises: Obtain the original time series data of the object to be predicted and perform preprocessing; Split the preprocessed time series data into several subsequences; An autoregressive transformer encoder module is used to capture the temporal dependencies of the subsequences, and a diffusion process is introduced in the encoding process to generate a self-supervisory signal; Combined with the self-supervisory signal, a denoising autoregressive transformer decoder module is used to fuse global and local features using an inverse diffusion process and a cross-attention mechanism to generate a deep representation; The depth representation is transformed in dimension through a linear connection layer to reconstruct an approximate representation of the original time series data as a prediction result.

2. The time series prediction method based on diffuse autoregressive transformer according to claim 1 is characterized in that: The obtaining of the original time series data of the object to be predicted and preprocessing the data includes: Get the multivariate original time series data X = [x1, x2, ..., x L ],X∈R C×L , where C represents the number of variables and L represents the length of the lookback window; After normalization, the normalized expression is: Among them, x c,i represents the i-th time series data on the c-th window, represents the mean of all samples on this dimension, Represents the standard deviation of all samples on this dimension.

3. The time series prediction method based on diffuse autoregressive transformer according to claim 2 is characterized in that: The preprocessed time series data is divided into several subsequences, including: The normalized time series data is divided into several segments using the fragment embedding technology. Each segment consists of data points at adjacent time steps. After segmentation, the time series data x 1:L Convert to x 1:N , where L is the product of N and the fragment length; Determine the length of each segment based on a preset time window; Through embedding technology, each fragment is mapped to a high-dimensional embedding space to obtain a subsequence z 1:N , as follows: Z 1:N =Embedding(x 1:N )。 4. The time series prediction method based on diffuse autoregressive transformer according to claim 3 is characterized in that: The method of using an autoregressive transformer encoder module to capture the temporal dependencies of the subsequences and introducing a diffusion process to generate a self-supervisory signal during the encoding process includes: Construct an encoder module based on the autoregressive transformer architecture to As input, the temporal dependencies of each fragment in the subsequence are captured through a multi-head attention mechanism, as follows: Where SOS represents the sequence start marker, PE represents the position encoding, and adding the position encoding makes the sequence have position features to capture the temporal dependencies between each segment; f() represents the encoder; Introducing a diffusion process into the encoded subsequence, gradually spreading to each time step Add noise to the encoded features to generate a self-supervisory signal Where j = 1, ..., N, the noise is controlled by the cosine scheduler to change its injection intensity and time step, as shown in the following formula: in α s =1-β s , β s Represents the noise coefficient with a diffusion step size of s, set it as an increasing sequence, and use the cosine scheduling strategy 5. The time series prediction method based on diffuse autoregressive transformer according to claim 4 is characterized in that: The self-supervisory signal is combined with the denoising autoregressive transformer decoder module to fuse global and local features using an inverse diffusion process and a cross attention mechanism to generate a deep representation, including: Construct a decoder module based on the denoising autoregressive transformer architecture, take the self-supervisory signal as the target signal, guide the gradual removal of the added noise through the inverse diffusion process, and use the cross-attention mechanism to fuse the global information of the encoding stage with the local features of the current decoding stage to generate the deep representation L1. The calculation formula is as follows: Where E is the expectation, represents the original distribution of the input time series, ∈ represents the noise added at each step, which satisfies the standard Gaussian distribution N(0,1).

6. The time series prediction method based on diffuse autoregressive transformer according to claim 5 is characterized in that: The process of transforming the dimension of the depth representation through the linear connection layer and reconstructing an approximate representation of the original time series data as a prediction result includes: The depth representation is transformed through a linear connection layer to match the dimension of the original time series data. The calculation formula is as follows: Where f represents the output of the representation network, and g represents the output of the decoder module and the linear connection layer; Reconstruct an approximate representation of the original time series, and output the approximate representation as the prediction result. The L2 calculation formula of the reconstructed approximate representation is as follows:

7. A time series prediction system based on diffuse autoregressive transformer, characterized in that: Applied to executing the time series prediction method according to any one of claims 1 to 6, the system comprises: The data preprocessing module is used to obtain the original time series data of the object to be predicted and perform preprocessing; Data partitioning module, used to split the preprocessed time series data into several subsequences; An encoder module, used for capturing the temporal dependency of the subsequences using an autoregressive transformer encoder module, and introducing a diffusion process in the encoding process to generate a self-supervisory signal; A decoder module for combining the self-supervisory signal to generate a deep representation by fusing global and local features using an inverse diffusion process and a cross-attention mechanism through a denoising autoregressive transformer decoder module; The prediction output module is used to transform the dimension of the depth representation through a linear connection layer and reconstruct an approximate representation of the original time series data as a prediction result.

Images