Time sequence prediction method and system of unconditional diffusion model
By introducing a non-conditional diffusion model in time series prediction, combining technical means such as conditional non-autoregressive diffusion model, dual encoder unit and historically irrelevant guiding unit, the existing methods have solved the problems of capturing long-term historical patterns and complex time dependencies, high computational cost and insufficient flexibility, and achieved more efficient and accurate time series prediction.
Patent Information
- Application Number
- CN202411898700.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2025-04-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing time series prediction method based on diffusion model is difficult to effectively capture long-term historical patterns and complex time dependencies, the calculation cost is high, the conversion is not flexible enough, and the processing capacity of non-stationary time series is limited.
A time series prediction method for non-conditional diffusion model is proposed, including conditional non-autoregressive diffusion model, dual encoder unit, historically independent guidance unit, conditional and unconditional generation unit, prediction unit and model training and optimization unit, through these components, sequence reconstruction, information extraction, denoising process guidance and prediction result output are realized.
It significantly improves prediction accuracy, reduces mean square error and continuous sorting probability scores, improves the model's adaptability to complex non-stationary time series, reduces calculation costs, and enhances the flexibility and applicability of the model.
Smart Images
Figure CN119830233A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of time prediction, and in particular to a time series prediction method and system of an unconditional diffusion model. Background Art
[0002] In recent years, time series forecasting has been widely used in finance, meteorology, energy and other fields. With the development of machine learning technology, time series forecasting methods based on diffusion models have gradually attracted the attention of researchers. This type of method generates forecast results by simulating the gradual denoising process of data from noise to target distribution.
[0003] However, existing time series forecasting methods based on diffusion models still have the following limitations:
[0004] (1) It is difficult to effectively capture long-term historical patterns and complex time dependencies, which is crucial for accurately simulating data distribution;
[0005] (2) Reliance on RNN or Transformer architectures results in high computational costs. Due to the iterative nature of the diffusion process, these complex neural network structures need to be run multiple times during the prediction process, which increases the computational complexity.
[0006] (3) The conversion between conditional generation and unconditional generation is not flexible enough. Existing methods are usually optimized for specific types of prediction tasks and are difficult to be applied efficiently in different prediction scenarios.
[0007] (4) The processing capability for non-stationary time series is limited, and it is difficult to adapt to rapidly changing data patterns. Summary of the invention
[0008] In order to solve the problems raised in the background technology, the present invention proposes a time series prediction method and system of an unconditional diffusion model, and the specific technical solution is as follows:
[0009] In one aspect of the present invention, a time series prediction system of an unconditional diffusion model is provided, comprising:
[0010] Conditional non-autoregressive diffusion model for sequence reconstruction;
[0011] Dual encoder units for information extraction;
[0012] A history-independent guidance unit to guide the denoising process;
[0013] Conditional and unconditional generation units to handle the denoising process;
[0014] A prediction unit, used for outputting prediction results;
[0015] Model training and prioritization unit, used for model training and verification;
[0016] An integration unit that integrates a conditional non-autoregressive diffusion model, a dual encoder unit, conditional and unconditional generation units, a prediction unit, and a model training and prioritization unit to achieve time series forecasting.
[0017] Preferably, the conditional non-autoregressive diffusion model comprises the following steps:
[0018] Introducing conditional information into the generation process, and using a non-autoregressive approach to generate the entire sequence at once;
[0019] The forward diffusion process is constructed by gradually adding noise to the target sequence;
[0020] Sequence reconstruction is achieved through a back-diffusion process;
[0021] The forward diffusion process is based on the Markov chain structure and is implemented by gradually adding Gaussian noise to the original sequence; the process is described by the following mathematical expression:
[0022]
[0023] The reparameterization technique is used to simplify the calculation:
[0024]
[0025] in, represents the cumulative noise factor, which is calculated by a predefined noise schedule;
[0026] The reverse denoising process trains a neural network to predict and remove the noise added during the diffusion process. The process includes the following elements:
[0027] First, build the prediction network:
[0028] ∈ θ (x t ,t,c)
[0029] Then define the loss function:
[0030]
[0031] Finally, denoising is achieved through iterative optimization:
[0032]
[0033] Preferably, the dual encoder unit comprises a sequence encoder: for extracting information representation from historical data with linear time complexity based on a state space model;
[0034] Temporal encoder: used to capture key temporal dependencies and extract temporal dynamic features.
[0035] Preferably, the history-independent guiding unit comprises:
[0036] Fusing the representations extracted by the sequence encoder and the temporal encoder;
[0037] Design a guided gating mechanism to randomly discard conditional information;
[0038] The processed conditional information is input into the denoising network to guide the denoising process.
[0039] Preferably, the conditional and unconditional generation units include:
[0040] Through the HFG mechanism, the denoising model can be flexibly switched between conditional generation and unconditional generation;
[0041] Enhance the ability of denoising models to capture complex time series patterns;
[0042] The expert denoiser is responsible for predicting the noise added in each diffusion step and gradually recovering the original signal;
[0043] By using multiple expert denoisers, the denoising model capacity can be increased without increasing the sampling complexity at each time step.
[0044] The expert denoisers at each stage work together to handle the entire denoising process from high noise to low noise.
[0045] Preferably, the prediction unit comprises:
[0046] Input historical time series data;
[0047] Extract feature representation through dual encoder;
[0048] The HFG mechanism is used to guide the denoising network to perform iterative prediction;
[0049] Output the prediction results for future time points.
[0050] Preferably, the model training and optimization unit includes:
[0051] The mean square error and continuous ranking probability score indicators were used to evaluate the model performance;
[0052] Model training and validation on large-scale multivariate time series datasets.
[0053] Preferably, the integrated unit includes an integrated conditional non-autoregressive diffusion model, a dual encoder unit, a conditional and unconditional generation unit, a prediction unit, and a model training and priority unit to achieve time series prediction.
[0054] A time series prediction method for an unconditional diffusion model comprises the following steps:
[0055] S1: Sequence reconstruction using conditional non-autoregressive diffusion model;
[0056] S2: Information extraction using dual encoder units;
[0057] S3: Using history-independent guidance units to guide the denoising process;
[0058] S4: denoising process using conditional and unconditional generation units;
[0059] S5: Output prediction results using the prediction unit;
[0060] S6: Model training and validation using model training and priority units;
[0061] S7: Using ensemble units to integrate conditional non-autoregressive diffusion models, dual encoder units, conditional and unconditional generation units, prediction units, and model training and prioritization units for time series prediction
[0062] Beneficial effects:
[0063] 1. The prediction accuracy is significantly improved. Compared with the best existing method, the mean square error (MSE) is reduced by an average of 11.44%, and the continuous ranking probability score (CRPS) is reduced by an average of 22.28%. These data show that the present invention has made significant progress in prediction accuracy and provides more reliable prediction results for practical applications.
[0064] 2. Adaptation to complex non-stationary time series: Through the combination of the conditional non-autoregressive diffusion model and the history-free bootstrapping (HFG) mechanism, the present invention can effectively capture the intrinsic patterns in complex, non-stationary time series and improve the model's adaptability to changing trends.
[0065] 3. Improved computational efficiency: The state-space based encoder is used to achieve feature extraction with linear time complexity. Compared with the traditional RNN or Transformer architecture, the computational cost is greatly reduced, enabling the model to process large-scale time series data more efficiently.
[0066] 4. Enhanced flexibility: The HFG mechanism enables the model to switch seamlessly between conditional generation and unconditional generation, enhancing the applicability of the model. The non-autoregressive structure enables the model to adapt to a variety of model architectures, improving the versatility of the model.
[0067] 5. Wide applicability: Experimental results on six real-world datasets show that the present invention performs well in different fields and different types of time series prediction tasks, demonstrating good generalization ability.
[0068] 6. Robustness improvement: By fusing the features extracted by the sequence encoder and the time encoder and combining the HFG mechanism, the present invention improves the model's resistance to noise and outliers and enhances the stability of the prediction results.
[0069] 7. Enhanced interpretability: The dual encoder structure enables the model to capture historical patterns and temporal dynamics separately, providing better interpretability for prediction results and helping users understand and trust the model’s predictions.
[0070] 8. Significant practical value: In time series forecasting tasks in multiple fields such as finance, meteorology, and energy, the method of the present invention has demonstrated performance superior to that of existing technologies, providing more reliable support for decision-making and risk management in these fields. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] Figure 1 It is the overall structural diagram of the present invention;
[0072] Figure 2 This is the experimental effect of the present invention Figure 1 ;
[0073] Figure 3 This is the experimental effect of the present invention Figure 2 ;
[0074] Figure 4 is a graph of ablation experiment data performed in the present invention; DETAILED DESCRIPTION
[0075] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.
[0076] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0077] Example 1: Figure 1-4 As shown, the present invention provides a time series prediction system of an unconditional diffusion model, comprising:
[0078] Conditional non-autoregressive diffusion model for sequence reconstruction;
[0079] Dual encoder units for information extraction;
[0080] A history-independent guidance unit to guide the denoising process;
[0081] Conditional and unconditional generation units to handle the denoising process;
[0082] A prediction unit, used for outputting prediction results;
[0083] Model training and prioritization unit, used for model training and verification;
[0084] An integration unit that integrates a conditional non-autoregressive diffusion model, a dual encoder unit, conditional and unconditional generation units, a prediction unit, and a model training and prioritization unit to achieve time series forecasting.
[0085] Specifically, the conditional non-autoregressive diffusion model includes the following steps:
[0086] Introducing conditional information into the generation process, and using a non-autoregressive approach to generate the entire sequence at once;
[0087] The forward diffusion process is constructed by gradually adding noise to the target sequence;
[0088] Sequence reconstruction is achieved through a back-diffusion process;
[0089] The forward diffusion process is based on the Markov chain structure and is implemented by gradually adding Gaussian noise to the original sequence; the process is described by the following mathematical expression:
[0090]
[0091] The reparameterization technique is used to simplify the calculation:
[0092]
[0093] in, represents the cumulative noise factor, which is calculated by a predefined noise schedule;
[0094] The reverse denoising process trains a neural network to predict and remove the noise added during the diffusion process. The process includes the following elements:
[0095] First, build the prediction network:
[0096] ∈ θ (x t ,t,c)
[0097] Then define the loss function:
[0098]
[0099] Finally, denoising is achieved through iterative optimization:
[0100]
[0101] Specifically, the dual encoder unit includes a sequence encoder: used to extract information representation from historical data with linear time complexity based on a state space model;
[0102] Temporal encoder: used to capture key temporal dependencies and extract temporal dynamic features.
[0103] Specifically, the state space model in the present invention is the Mamba2 model. Mamba2 adopts a discrete state space model, and its core mathematical expression is:
[0104] State update equation:
[0105] x t+1 =A(Δ t )x t +B(Δ t ) t
[0106] Observation equation:
[0107] y t =Cx t +Du t
[0108] Where: x t represents the state vector at time t, u t represents the input sequence, y t represents the output sequence, Δ t represents the time step, A(Δ t ),B(Δ t ), C, D are learnable parameter matrices.
[0109] Among them, time series prediction is to predict future data based on past data. For example, in weather forecasting, the weather in the past seven days is used to predict the weather in the next three days. Here, "the weather data in the past seven days" is historical data.
[0110] Among them, the information representation extracted from historical data aims to extract important features to guide the subsequent forecasting work, which may include level, trend, seasonality and other characteristics;
[0111] Among them, when the time encoder extracts temporal dynamic features, local scattered point smoothing estimation is used. The core steps and formulas for extracting time series features are as follows: For a time point t, first select window data points with a width of h around it, and assign a weight w to each data point i =(1-|x i -t|3 / h 3 ) 3 , where the farther the data point is from the center point t, the smaller the weight. Then a weighted least squares fit is performed within the window Get a local linear estimate. By moving the window point by point and repeating this process, we can finally get a smooth trend line for the entire time series. The advantage of this method is that it does not presuppose a global function form, but uses the information of local data to capture the dynamic characteristics of the sequence, so it can strike a good balance between flexibility and robustness. The window width h is a key parameter. A larger h will produce a smoother trend but may lose details, while a smaller h will retain more details but may overfit the noise.
[0112] Therefore, the time encoder extracts temporal dynamic features in order to extract important features to guide subsequent prediction work, including level, trend, seasonality and other features.
[0113] In summary, sequence reconstruction is the reverse denoising process mentioned above, which can capture the characteristics of the original data, such as level, trend and seasonality.
[0114] For example, in time series feature representation, it can automatically capture the periodic changes in the sequence, accurately extract long-term and short-term trend information, and retain important patterns such as seasonal fluctuations.
[0115] In the relational feature representation, it is possible to reveal the intrinsic dependencies between sequence elements, establish correlation mapping between features, and maintain the temporal continuity characteristics of the sequence.
[0116] Furthermore, the following support can also be provided for dual encoder units:
[0117] Quality improvement: reduce noise interference in raw data, supplement possible missing information, and optimize the overall distribution characteristics of data;
[0118] Feature enhancement: providing clear temporal dependency information, highlighting key patterns in sequences, and preserving valuable contextual associations.
[0119] Furthermore, in the present invention, the time encoder extracts temporal dynamic features using local scattered point smoothing estimation, and the core steps and formulas for extracting time series features are as follows: For a time point t, firstly, a window data point with a width of h is selected around it, and a weight w is assigned to each data point. i =(1-|x i -t| 3 / h 3 ) 3 , where the farther the data point is from the center point t, the smaller the weight. Then a weighted least squares fit is performed within the window Get a local linear estimate. By moving the window point by point and repeating this process, we can finally get a smooth trend line for the entire time series. The advantage of this method is that it does not presuppose a global function form, but uses the information of local data to capture the dynamic characteristics of the sequence, so it can strike a good balance between flexibility and robustness. The window width h is a key parameter. A larger h will produce a smoother trend but may lose details, while a smaller h will retain more details but may overfit the noise.
[0120] Therefore, the temporal encoder extracts temporal dynamic features to guide subsequent prediction work, including level, trend, seasonality and other features.
[0121] Specifically, the history-independent guidance units include:
[0122] Fusing the representations extracted by the sequence encoder and the temporal encoder;
[0123] Design a guided gating mechanism to randomly discard conditional information;
[0124] The processed conditional information is input into the denoising network to guide the denoising process;
[0125] Specifically, the representation extracted by the fused sequence encoder and the time encoder adopts a weighted linear structure, such as input = w1*input1+w2*input2, in order to input this information into the denoising network, and the guided gating mechanism directly sets the input to a zero matrix (input = 0) with a probability of 0.1, thereby increasing the robustness of the model and the ability of zero-shot learning, wherein the information extracted before the conditional information package is randomly discarded; and the processed information specifically refers to the fused information.
[0126] Specifically, the conditional and unconditional generation units include:
[0127] Through the HFG mechanism, the denoising model can be flexibly switched between conditional generation and unconditional generation;
[0128] Enhance the ability of denoising models to capture complex time series patterns;
[0129] The expert denoiser is responsible for predicting the noise added in each diffusion step and gradually recovering the original signal;
[0130] By using multiple expert denoisers, the denoising model capacity can be increased without increasing the sampling complexity at each time step.
[0131] The expert denoisers at each stage work together to handle the entire denoising process from high noise to low noise.
[0132] In the predicted time series, the process is as follows: the original data is denoised to obtain input 1, the original data is passed through a model to obtain the information extracted from the historical data, namely input 2, the original data is passed through a time encoder to obtain input 3, and the three inputs are passed through another model to obtain the final predicted time series. (Here, input 2 and input 3 will go through a weighted and randomly discarded process)
[0133] If input 2 and input 3 are discarded, the matrix becomes zero, which is unconditional generation. If they are not discarded, it is conditional generation. If input 2 and input 3 are discarded, the historical information is lost, which is historical-free guidance (HFG).
[0134] Among them, the purpose of the model to capture complex time series patterns is to increase the robustness of the denoising model and the ability of zero-shot learning by randomly discarding conditional information;
[0135] Complex time series patterns include many aspects, such as large time span, complex patterns (not simple sinusoidal or exponential types), and many features;
[0136] Among them, the expert denoiser is the denoising model, which is part of the non-autoregressive diffusion model;
[0137] Among them, the original signal is the true value.
[0138] Specifically, the prediction unit includes
[0139] Input historical time series data;
[0140] Extract feature representation through dual encoder;
[0141] The HFG mechanism is used to guide the denoising network to perform iterative prediction;
[0142] Output the prediction results for future time points.
[0143] Specifically, the original data is denoised to obtain input 1, the original data is passed through a model to obtain the information extracted from the historical data, namely input 2, the original data is passed through a time encoder to obtain input 3, and the three inputs are passed through a denoising model to obtain the final predicted time series. (Here, input 2 and input 3 will first undergo a weighted and randomly discarded process)
[0144] Specifically, the model training and optimization unit includes:
[0145] The mean square error (MSE) and continuous ranked probability score (CRPS) indicators were used to evaluate the model performance;
[0146] Model training and validation using large-scale multivariate time series datasets;
[0147] Specifically, the integrated unit includes an integrated conditional non-autoregressive diffusion model, a dual encoder unit, a conditional and unconditional generation unit, a prediction unit, and a model training and priority unit to achieve time series prediction.
[0148] Embodiment 2: A time series prediction method of an unconditional diffusion model comprises the following steps:
[0149] S1: Sequence reconstruction using conditional non-autoregressive diffusion model;
[0150] S2: Information extraction using dual encoder units;
[0151] S3: Using history-independent guidance units to guide the denoising process;
[0152] S4: denoising process using conditional and unconditional generation units;
[0153] S5: Output prediction results using the prediction unit;
[0154] S6: Use model training and priority units to train and validate the model;
[0155] S7: Using ensemble units to integrate conditional non-autoregressive diffusion models, dual encoder units, conditional and unconditional generation units, prediction units, and model training and prioritization units for time series forecasting.
[0156] In summary, if Figure 1 As shown, it is a flow chart of the present invention, wherein Y k is the noise-added time series input 1,X ～ and z are input 2 and input 3, ke is the current denoising time step, and the denoising network needs to go through k denoising times to get the predicted value;
[0157] like Figure 2 As shown, the first column is the name of the dataset. The second column is the prediction length; MSE and CRPS are evaluation indicators; the first row is the model name; for example, the experimental process is to input the original data with a length of 16, obtain the predicted value with a length of 16, and compare it with the true value to obtain the evaluation indicator. Here, the input and output are set to be equal in length during evaluation. Of course, the input and output of the model can also be of different lengths. Generally, the longer the input, the better the model effect;
[0158] like Figure 3 As shown in the figure, it is a visual demonstration. The left side is the model input, the red value on the right is the predicted value of the model output, and the blue is the true value. The more similar the red line is to the blue line, the more accurate the model prediction is.
[0159] like Figure 4As shown in the figure, 16 / 16 represents the input and output lengths. The second and third rows are the architectures of the denoising network. We used the Mamba2 architecture, which is better than the MLP architecture and the Transformer architecture. S is the sequence encoder, T is the temporal encoder, and the last G is the random drop mechanism.
[0160] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. A time series prediction system for an unconditional diffusion model, characterized by: include: Conditional non-autoregressive diffusion model for sequence reconstruction; Dual encoder units for information extraction; A history-independent guidance unit to guide the denoising process; Conditional and unconditional generation units to handle the denoising process; A prediction unit, used for outputting prediction results; Model training and prioritization unit, used for model training and verification; An integration unit that integrates a conditional non-autoregressive diffusion model, a dual encoder unit, conditional and unconditional generation units, a prediction unit, and a model training and prioritization unit to achieve time series forecasting.
2. A time series prediction system of an unconditional diffusion model as claimed in claim 1, characterized in that: The conditional non-autoregressive diffusion model includes the following steps: Introducing conditional information into the generation process, and using a non-autoregressive approach to generate the entire sequence at once; The forward diffusion process is constructed by gradually adding noise to the target sequence; Sequence reconstruction is achieved through a back-diffusion process; The forward diffusion process is based on the Markov chain structure and is implemented by gradually adding Gaussian noise to the original sequence; the process is described by the following mathematical expression: The reparameterization technique is used to simplify the calculation: in, represents the cumulative noise factor, which is calculated by a predefined noise schedule; The reverse denoising process trains a neural network to predict and remove the noise added during the diffusion process. The process includes the following elements: First, build the prediction network: ∈ θ (x t ,t,c) Then define the loss function: Finally, denoising is achieved through iterative optimization:
3. A time series prediction system of an unconditional diffusion model as claimed in claim 1, characterized in that: The dual encoder unit includes a sequence encoder: used to extract information representation from historical data with linear time complexity based on a state space model; Temporal encoder: used to capture key temporal dependencies and extract temporal dynamic features.
4. A time series prediction system of an unconditional diffusion model as claimed in claim 1, characterized in that: History-independent boot units include: Fusing the representations extracted by the sequence encoder and the temporal encoder; Design a guided gating mechanism to randomly discard conditional information; The processed conditional information is input into the denoising network to guide the denoising process.
5. The time series prediction system of the unconditional diffusion model according to claim 1, characterized in that: Conditional and unconditional generation units include: Through the HFG mechanism, the denoising model can be flexibly switched between conditional generation and unconditional generation; Enhance the ability of denoising models to capture complex time series patterns; The expert denoiser is responsible for predicting the noise added in each diffusion step and gradually recovering the original signal; By using multiple expert denoisers, the denoising model capacity can be increased without increasing the sampling complexity at each time step. The expert denoisers at each stage work together to handle the entire denoising process from high noise to low noise.
6. A time series prediction system of an unconditional diffusion model as claimed in claim 1, characterized in that: The prediction unit comprises: Input historical time series data; Extract feature representation through dual encoder; The HFG mechanism is used to guide the denoising network to perform iterative prediction; Output the prediction results for future time points.
7. A time series prediction system of an unconditional diffusion model as claimed in claim 1, characterized in that: The model training and optimization unit includes: The mean square error and continuous ranking probability score indicators were used to evaluate the model performance; Model training and validation on large-scale multivariate time series datasets.
8. A time series prediction system of an unconditional diffusion model as claimed in claim 1, characterized in that: The integrated unit includes an integrated conditional non-autoregressive diffusion model, a dual encoder unit, a conditional and unconditional generation unit, a prediction unit, and a model training and priority unit to achieve time series prediction.
9. A time series prediction method for an unconditional diffusion model, characterized by: The steps include: S1: Sequence reconstruction using conditional non-autoregressive diffusion model; S2: Information extraction using dual encoder units; S3: Using history-independent guidance units to guide the denoising process; S4: denoising process using conditional and unconditional generation units; S5: Output prediction results using the prediction unit; S6: Use model training and priority units to train and validate the model; S7: Using ensemble units to integrate conditional non-autoregressive diffusion models, dual encoder units, conditional and unconditional generation units, prediction units, and model training and prioritization units for time series forecasting.