Multi-element time series prediction method and system based on flexible actor-commentator
By employing a flexible actor-critic model, combining multilayer perceptron and Fourier decomposition to extract features, and utilizing reinforcement learning to optimize the policy network, the shortcomings of traditional time series forecasting methods in handling nonlinear and non-stationary data are addressed, achieving efficient and stable multivariate time series forecasting.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-08
- Publication Date
- 2026-04-10
AI Technical Summary
Existing time series forecasting methods perform poorly when dealing with nonlinear and non-stationary data, struggle to capture multidimensional time series features, are easily affected by outliers, and have insufficient model generalization ability.
We employ a flexible actor-critic multivariate time series prediction method, extracting trend, seasonal, and periodic features through multilayer perceptron and Fourier decomposition. We then combine reinforcement learning to construct a flexible actor-critic model and utilize policy network, value network, and target value network for soft updates and optimization, achieving efficient feature construction and prediction.
With limited computing resources, the model's stability and robustness are improved, overfitting is avoided, and the efficiency and generalization ability of multivariate time series forecasting are enhanced.
Smart Images
Figure CN121835336A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent agent technology, and in particular to a multivariate time series prediction method and system based on flexible actor-critic. Background Technology
[0002] With the rapid advancement of the information revolution and the swift development of technology, scientific and technological thinking, innovation, and applications are changing the original competitive landscape and ecosystem of the internet and finance industries. Under this new situation, the rapidly developing technologies such as the internet, big data, and artificial intelligence are reshaping the model and path of financial development, and the role, status, and development of time series forecasting are becoming increasingly important in various fields.
[0003] Traditional time series forecasting methods commonly include moving averages, exponential smoothing, and ARIMA models. These methods can predict future trends by fitting historical data, offering a certain level of accuracy and interpretability. However, they often require stationarity checks on the data, perform poorly in handling nonlinear or non-stationary conditions, have limited ability to capture categorical features and high-dimensional data features, and struggle with multidimensional time series forecasting. They are also susceptible to outliers and exhibit poor model generalization ability.
[0004] Machine learning methods handle more complex data and do not require excessive preprocessing, but they require more computational resources and time for training. They have high requirements for the selection and adjustment of model parameters, poor model interpretability, high requirements for data quality, and are prone to overfitting, resulting in poor time series prediction performance.
[0005] Therefore, the existing technology still needs further development. Summary of the Invention
[0006] To address the aforementioned technical problems, this invention provides a method and system for multivariate time series prediction based on flexible actor-critic, which can solve the technical problem of poor time series prediction performance in existing time series prediction algorithms.
[0007] A first aspect of this invention provides a multivariate time series prediction method based on flexible actor-critic, the method comprising:
[0008] Historical data is acquired, preprocessed, and then feature extraction is performed on the preprocessed historical data to complete the construction of historical features. The feature extraction operation is based on multilayer perceptron and Fourier decomposition to capture trend features, seasonal features, periodic features, and holiday features in the historical data.
[0009] A reinforcement learning element for multivariate time series prediction is defined, and a flexible actor-critic model is constructed based on the reinforcement learning element. The flexible actor-critic model includes a policy network, two value networks, and two target value networks. The flexible actor-critic model achieves soft updates and optimizations of the policy and value functions by minimizing the flexible value function and entropy regularization, as well as the importance sampling algorithm and target networks.
[0010] The flexible actor-critic model is trained based on the historical features, and after training, a multivariate time series prediction model is obtained.
[0011] The system acquires real-time data to be predicted, preprocesses the real-time data, extracts features from the preprocessed real-time data to construct real-time data features, inputs the real-time data features into the multivariate time series prediction model, obtains the output of the multivariate time series prediction model, and completes the prediction of future time series.
[0012] Optionally, the step of acquiring historical data and preprocessing the historical data includes:
[0013] Based on specific business processes or domain knowledge, historical data is obtained, and the historical data is the true value of the time series.
[0014] After performing data cleaning and normalization operations on the historical data, preprocessed historical data is obtained.
[0015] Optionally, after performing data cleaning and normalization operations on the historical data to obtain preprocessed historical data, the method further includes:
[0016] Based on the preprocessed historical data, time-series index features, statistical index features, volatility index features, slope index features, and regression index features are calculated.
[0017] Calculate the variance inflation factor among the above-mentioned indicator features, and remove features whose variance factor value is greater than a preset threshold.
[0018] Calculate the Pearson correlation coefficient between the above-mentioned indicator features and the predicted values, and take the features with a Pearson correlation coefficient greater than the preset value as the target historical features.
[0019] Optionally, feature extraction is performed on the preprocessed historical data to construct historical features, including:
[0020] A nonlinear model of a multilayer perceptron is constructed, and the trend characteristics of time series are captured based on the nonlinear model of the multilayer perceptron.
[0021] A first linear model based on Fourier series is constructed, and the seasonal and periodic features of the time series are captured based on the first linear model;
[0022] A second linear model is constructed, and the holiday features of the time series are captured based on the second linear model;
[0023] By fusing the trend features, seasonal features, cyclical features, and holiday features, the feature extraction of the time series is completed, and the construction of historical features is achieved.
[0024] Optionally, reinforcement learning elements for multivariate time series prediction are defined, and a flexible actor-critic model is constructed based on these elements. The flexible actor-critic model includes a policy network, two value networks, and two target value networks, comprising:
[0025] The state elements, action elements, reward elements, and strategy elements of the multivariate time series prediction are defined.
[0026] Construct a policy network, which outputs an action policy based on the current environmental state;
[0027] Construct two value networks, which are used to combine the value obtained by performing actions in the current state with the current state and actions.
[0028] Construct two target value networks, which are used to estimate the target value of state-action pairs under the current policy.
[0029] A second aspect of this invention provides a flexible actor-critic multivariate time series prediction system, the system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, performs the following steps:
[0030] Historical data is acquired, preprocessed, and then feature extraction is performed on the preprocessed historical data to complete the construction of historical features. The feature extraction operation is based on multilayer perceptron and Fourier decomposition to capture trend features, seasonal features, periodic features, and holiday features in the historical data.
[0031] A reinforcement learning element for multivariate time series prediction is defined, and a flexible actor-critic model is constructed based on the reinforcement learning element. The flexible actor-critic model includes a policy network, two value networks, and two target value networks. The flexible actor-critic model achieves soft updates and optimizations of the policy and value functions by minimizing the flexible value function and entropy regularization, as well as the importance sampling algorithm and target networks.
[0032] The flexible actor-critic model is trained based on the historical features, and after training, a multivariate time series prediction model is obtained.
[0033] The system acquires real-time data to be predicted, preprocesses the real-time data, extracts features from the preprocessed real-time data to construct real-time data features, inputs the real-time data features into the multivariate time series prediction model, obtains the output of the multivariate time series prediction model, and completes the prediction of future time series.
[0034] Optionally, when the computer program is executed by the processor, it performs the following steps:
[0035] Based on specific business processes or domain knowledge, historical data is obtained, and the historical data is the true value of the time series.
[0036] After performing data cleaning and normalization operations on the historical data, preprocessed historical data is obtained.
[0037] Optionally, when the computer program is executed by the processor, it further performs the following steps:
[0038] Based on the preprocessed historical data, time-series index features, statistical index features, volatility index features, slope index features, and regression index features are calculated.
[0039] Calculate the variance inflation factor among the above-mentioned indicator features, and remove features whose variance factor value is greater than a preset threshold.
[0040] Calculate the Pearson correlation coefficient between the above-mentioned indicator features and the predicted values, and take the features with a Pearson correlation coefficient greater than the preset value as the target historical features.
[0041] Optionally, when the computer program is executed by the processor, it further performs the following steps:
[0042] A nonlinear model of a multilayer perceptron is constructed, and the trend characteristics of time series are captured based on the nonlinear model of the multilayer perceptron.
[0043] A first linear model based on Fourier series is constructed, and the seasonal and periodic features of the time series are captured based on the first linear model;
[0044] A second linear model is constructed, and the holiday features of the time series are captured based on the second linear model;
[0045] By fusing the trend features, seasonal features, cyclical features, and holiday features, the feature extraction of the time series is completed, and the construction of historical features is achieved.
[0046] A third aspect of the present invention provides a non-volatile computer-readable storage medium, characterized in that the non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform the above-described flexible actor-critic multivariate time series prediction method.
[0047] The technical solution provided in this invention involves acquiring historical data, preprocessing the historical data, and performing feature extraction to construct historical features; defining reinforcement learning elements for multivariate time series prediction, and constructing a flexible actor-critic model based on these elements; training the flexible actor-critic model based on historical features to obtain a multivariate time series prediction model; acquiring real-time data to be predicted, preprocessing the real-time data, and performing feature extraction to construct real-time data features; inputting the real-time data features into the multivariate time series prediction model to obtain the output results and complete the prediction of future time series. This invention enables efficient learning and optimization in large-scale environments with limited computing resources, avoids overfitting problems, and greatly improves the model's stability, robustness, and generalization ability, thereby increasing the efficiency of multivariate time series prediction. Attached Figure Description
[0048] Figure 1 This is a flowchart illustrating an embodiment of a multivariate time series prediction method based on flexible actor-criticism in this invention.
[0049] Figure 2 This is a schematic diagram of the structure of a flexible actor-critic model, which is an embodiment of a flexible actor-critic multivariate time series prediction method based on flexible actor-critic.
[0050] Figure 3 This is a schematic diagram of the network structure of a strategy network in an embodiment of a flexible actor-critic multivariate time series prediction method according to an embodiment of the present invention;
[0051] Figure 4 This is a schematic diagram of the network structure of a value network in an embodiment of a flexible actor-critic multivariate time series prediction method according to an embodiment of the present invention;
[0052] Figure 5 This is a flowchart illustrating the network training process of a flexible actor-critic model, representing an embodiment of a multivariate time series prediction method based on the flexible actor-critic model.
[0053] Figure 6 This is a schematic diagram of the hardware structure of another embodiment of a multivariate time series prediction system based on flexible actor-criticism in this invention. Detailed Implementation
[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0055] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0056] Please see Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of a multivariate time series prediction method based on flexible actor-criticism in this invention. Figure 1 As shown, it includes:
[0057] Step S100: Obtain historical data, preprocess the historical data, perform feature extraction on the preprocessed historical data, and complete the construction of historical features. The feature extraction operation is based on multilayer perceptron and Fourier decomposition to capture trend features, seasonal features, periodic features and holiday features in the historical data.
[0058] Step S200: Define reinforcement learning elements for multivariate time series prediction, and construct a flexible actor-critic model based on the reinforcement learning elements. The flexible actor-critic model includes a policy network, two value networks, and two target value networks. The flexible actor-critic model achieves soft updates and optimizations of the policy and value functions by minimizing the flexible value function and entropy regularization, as well as the importance sampling algorithm and target networks.
[0059] Step S300: Train the flexible actor-critic model based on the historical features. After training, a multivariate time series prediction model is obtained.
[0060] Step S400: Obtain the real-time data to be predicted, preprocess the real-time data, perform feature extraction on the preprocessed real-time data to complete the construction of real-time data features, input the real-time data features into the multivariate time series prediction model, obtain the output results of the multivariate time series prediction model, and complete the prediction of future time series.
[0061] In specific implementation, this embodiment of the invention acquires historical data, performs anomaly detection, missing value filling, and normalization processing; calculates fluctuation, overlap, and autoregressive slope indicators; constructs a multilayer perceptron, and combines Fourier decomposition to capture trend features, seasonal features, periodic features, and holiday features, thus completing the feature engineering construction.
[0062] We define the reinforcement learning elements for multivariate temporal prediction, construct a policy network, two value networks, and two target value networks. By minimizing the flexible value function and entropy regularization, and combining importance sampling techniques with the target networks, we achieve soft updates and optimizations of the policy and value functions.
[0063] Based on a preset ratio, the historical features are divided into a training set, a test set, and a validation set. The flexible actor-critic model is trained using the training set, and its effectiveness is verified using the validation set. Model parameters are adjusted based on the verification results. The adjusted flexible actor-critic model is then tested using the test set. When the test results meet the criteria, the flexible actor-critic model is selected, resulting in a multivariate time series prediction model. For example: based on historical data, maintaining the sequential nature of the sequence, the training set, test set, and validation set are divided in a 6:3:1 ratio; based on the data in the training set, the model parameters are updated by comparing the differences between the current strategy and historical strategies, completing model training; the trained model is saved, backtested, and its effectiveness is verified; the model parameters are adjusted, completing model selection.
[0064] The selected time series forecasting model is deployed to the production environment to make real-time predictions of future time series.
[0065] This invention acquires historical data and performs data preprocessing; constructs a multilayer perceptron and combines Fourier decomposition to capture trend features, seasonal features, periodic features, and holiday features; defines reinforcement learning elements for multivariate time series prediction, and builds a policy network, two value networks, and two target value networks. By minimizing the flexible value function and entropy regularization, and combining importance sampling techniques with the target networks, soft updates and optimizations of the policy and value functions are achieved for model training; backtesting is performed based on the trained model to verify its effectiveness, and models with strong fitting and generalization abilities are selected for real-time prediction of future time series. This invention enables efficient learning and optimization in large-scale environments with limited computing resources, avoids overfitting problems, and greatly improves the stability, robustness, and generalization ability of the model.
[0066] Furthermore, historical data is acquired, and the historical data is preprocessed, including:
[0067] Based on specific business processes or domain knowledge, historical data is obtained, and the historical data is the true value of the time series.
[0068] After performing data cleaning and normalization operations on the historical data, preprocessed historical data is obtained.
[0069] In practice, the true value of the time series is obtained based on specific business processes or domain knowledge.
[0070] Data cleaning operations include: deleting duplicate values from the original data summary; filling missing values with lag 7, lag 14 data, and linear interpolation; defining values smaller than a specific value and negative values as outliers and processing them with piecewise linear smoothing; and performing logarithmic processing on the data to remove heteroscedasticity from the sequence, reduce the impact of outliers, and increase the stationarity of the data.
[0071] The lag feature in time series refers to using data from a period of time preceding a given point in time as a feature to predict the value at that point in time. Lag features are typically widely used in time series forecasting because they are intuitive and interpretable, and can capture trends and seasonality in time series data.
[0072] In time series analysis, lag features include lag order 1, lag order 2, lag order 3, ..., lag order n, where n represents the number of time steps in the lag. For example, a lag order 7 feature means using data from the previous 7 time steps as features to predict the value at the current time step.
[0073] Lag features can capture the correlation and trend of time series. In this invention, lag 7 and lag 14 features are selected for filling in order to preserve the periodic effect of the time series, thereby obtaining the best prediction effect.
[0074] In regression analysis, outliers can easily cause the regression curve to deviate from the true trend, distorting the true relationship between variables and weakening model performance. Therefore, outlier processing is necessary. Winsorization is a commonly used data processing method that can reduce the impact of extreme values (outliers). The basic idea of winsorization is to replace values less than the p-th percentile (p% quantile) and greater than the q-th percentile (q% quantile) with their respective p-th and q-th percentile values. "Winsor 1%" in winsorization refers to replacing values less than the 1st percentile and greater than the 99th percentile with their respective 1% and 99th percentile values, essentially retaining 99% of the data and removing the smallest and largest 1%. This method is often used to handle data with a large number of extreme outliers, effectively reducing their impact and improving data stability and robustness. The specific value of this invention is 1%, based on Winsor's 1% tail reduction processing. Considering practical application scenarios such as peak holidays and other special time periods, the maximum value generated is also important information that needs to be predicted in time series. Therefore, only the value less than 1% is processed.
[0075] Normalization processing includes: normalizing time series data to obtain preprocessed historical data, thereby reducing the impact of data with different dimensions on feature weights and improving the convergence speed and stability of the model.
[0076] Furthermore, after performing data cleaning and normalization operations on the historical data to obtain preprocessed historical data, the process further includes:
[0077] Based on the preprocessed historical data, time-series index features, statistical index features, volatility index features, slope index features, and regression index features are calculated.
[0078] Calculate the variance inflation factor among the above-mentioned indicator features, and remove features whose variance factor value is greater than a preset threshold.
[0079] Calculate the Pearson correlation coefficient between the above-mentioned indicator features and the predicted values, and take the features with a Pearson correlation coefficient greater than the preset value as the target historical features.
[0080] In practice, time-series indicators, statistical indicators, volatility indicators, slope indicators, and regression indicators are calculated based on historical data. The variance inflation factor is used to detect multicollinearity among features, and features with high variance inflation factor values are eliminated to reduce the impact of collinearity. The Pearson correlation coefficient between features and predicted values is calculated as a standard to measure the effectiveness of features. The effectiveness of features is tested, and features with correlation values greater than 0.1 are selected to participate in model training.
[0081] The above-mentioned indicators can be collectively referred to as characteristic variables, which are mainly classified based on the nature and purpose of the characteristics.
[0082] Time-series indicators: These indicators are generally calculated based on time-series data and include, but are not limited to, the trends and seasonality of the time series. For example, the mean, variance, and autocorrelation of a time series are time-series indicators.
[0083] Statistical indicators: These indicators are generally derived from statistical analysis of time series data, including but not limited to percentiles, skewness, and kurtosis. These indicators can reflect the distribution characteristics of the time series.
[0084] Volatility indicators: These indicators are generally used to describe the volatility of time series data, including but not limited to volatility, standard deviation, and coefficient of variation. These indicators can be used to measure the risk level of time series data.
[0085] Slope-related indicators: These indicators generally describe the trend characteristics of time series, including but not limited to slope, trend line intercepts, etc. These indicators can be used to predict the future trend of time series.
[0086] Regression indicators: These indicators are generally used to describe the relationship between time series data and other variables, such as the relationship between time series data and macroeconomic indicators, meteorological data, etc. Common regression indicators include correlation coefficients, regression coefficients, and residuals.
[0087] Furthermore, feature extraction is performed on the preprocessed historical data to construct historical features, including:
[0088] A nonlinear model of a multilayer perceptron is constructed, and the trend characteristics of time series are captured based on the nonlinear model of the multilayer perceptron.
[0089] A first linear model based on Fourier series is constructed, and the seasonal and periodic features of the time series are captured based on the first linear model;
[0090] A second linear model is constructed, and the holiday features of the time series are captured based on the second linear model;
[0091] By fusing the trend features, seasonal features, cyclical features, and holiday features, the feature extraction of the time series is completed, and the construction of historical features is achieved.
[0092] In practice, a neural network is built that includes trend models, seasonality models, and holiday models to capture the trend, seasonality, periodicity, and holiday characteristics of time series.
[0093] Trend Model: A nonlinear model using a multilayer perceptron is constructed to capture the long-term trend of time series data. The input layer receives the timestamps and historical values of the time series, and the output layer outputs the predicted values. The hidden layers consist of multiple fully connected layers, each with multiple neurons. The activation function for each neuron is the ReLU function. By performing layer-by-layer propagation and nonlinear transformation on the input and output, the trend model can learn the nonlinear trend of the time series.
[0094] Seasonal Model: A linear model based on Fourier series is built to capture the seasonal and periodic characteristics of time series. By splitting the time series into multiple subsequences, performing a Fourier transform on each subsequence, selecting a specific number of Fourier coefficients as the input of the model, and predicting future values through a linear model (with a weight matrix), seasonal and periodic changes in the time series can be captured.
[0095] Holiday model: A linear model is built to capture the holiday effect in time series. It can represent the impact of special events (such as holidays) on time series. Each special event is encoded as a binary variable and multiplied with the output of the trend model and the seasonality model to finally obtain the predicted value of the time series.
[0096] Finally, the trend, seasonal, cyclical, and holiday features are fused together using gated linear units, and then the predicted value is output after passing through a fully connected layer.
[0097] Furthermore, reinforcement learning elements for multivariate time series prediction are defined, and a flexible actor-critic model is constructed based on these elements. The flexible actor-critic model includes a policy network, two value networks, and two target value networks, comprising:
[0098] The state elements, action elements, reward elements, and strategy elements of the multivariate time series prediction are defined.
[0099] Construct a policy network, which outputs an action policy based on the current environmental state;
[0100] Construct two value networks, which are used to combine the value obtained by performing actions in the current state with the current state and actions.
[0101] Construct two target value networks, which are used to estimate the target value of state-action pairs under the current policy.
[0102] In practice, the reinforcement learning elements in the time-series prediction process include, but are not limited to, the definitions of reinforcement learning elements such as state, action, and reward.
[0103] Definition of elements in temporal prediction reinforcement learning: such as Figure 2As shown, in reinforcement learning, an agent learns the optimal policy by interacting with its environment. In time series prediction, the environment refers to the current time series data and its associated features. At each time step, the agent selects an action, receives a reward, and then moves to the next time step, selecting another action in the new state. The agent's goal is to minimize prediction error and improve prediction accuracy by learning the optimal policy.
[0104] Time series prediction based on reinforcement learning includes the following elements:
[0105] State: All information observed by the agent at a specific point in time, including current historical observations and time series indicators such as time series indicators, statistical indicators, and volatility indicators, including seasonality, trend, periodicity, and other external variables related to the prediction target, used to predict the value at the next point in time.
[0106] Action: An agent's action is to output a definite value in a continuous action space based on the current state and the agent's policy. The definite value designed in this invention is a non-negative predicted value.
[0107] Reward: After the agent performs an action, the accuracy of the prediction is used as the reward or penalty for that action and is fed back to the agent.
[0108]
[0109] Where actual_value represents the actual value and predict_value represents the predicted value.
[0110] Policy: Specifies the action to be taken in each state to maximize long-term gains. It can be a deterministic or stochastic policy, and is typically represented as a function that takes states as input and outputs actions.
[0111] Based on the current state and reward, the reinforcement learning algorithm selects an action, outputs a predicted value, and updates the policy to improve the accuracy and stability of predictions. Through iterative learning, the algorithm can learn how to take the optimal predictive action in different states, thereby improving the accuracy and stability of time series predictions.
[0112] In reinforcement learning, an intelligent agent differs from intelligent robots and smartphones. An intelligent agent is an entity capable of observing its environment, making decisions, and learning through interaction with it. The agent's goal is to learn optimal behavioral strategies through interaction with the environment, maximizing its rewards. The agent can perceive the state of the environment and choose an action based on that state. The environment provides feedback on the agent's actions, evaluating them through a reward function. The agent adjusts its strategy based on the reward feedback to better adapt to the environment.
[0113] Furthermore, the flexible actor-critic model includes a policy network, two value networks, and two target value networks.
[0114] The policy network is a neural network used to generate actions. Based on the current environmental state, it outputs an action policy that maximizes the expected reward in the future. The value network, on the other hand, gives the value obtained by performing the action in the current state, based on the current state and the action. The target value network estimates the target value of the state-action pair under the current policy.
[0115] like Figure 3 As shown, the policy network comprises an input layer, three fully connected layers, four activation layers, a mean network, a variance network, and an output layer. The mean network and variance network output the logarithm of the average action value and the standard deviation of the action, which are used to calculate the loss function of the policy network and the probability distribution used to generate actions. Specifically, a Gaussian distribution is generated using the average action value and the standard deviation, and actions are then sampled from this distribution. Furthermore, to ensure that the sampled action values are within an acceptable range, the sampled values are scaled using a hyperbolic tangent function to limit their range to between -1 and 1.
[0116] like Figure 4 As shown, the network structures of the two target value networks and the two value networks are consistent, with one-to-one initialization parameters, including an input layer, two fully connected layers, two activation layers, and an output layer. The two value networks output two different estimates; the smaller of the two is used to approximate the value estimate, guiding policy improvement. The target value network is used to estimate the value of the next state and update the current policy network and value network based on this value. This approach addresses the error in value function estimation, providing a more stable and reliable value estimate for the next state, thereby improving the learning performance of the SAC algorithm.
[0117] The inputs to the target value network and the value network are different. The input to the value network is the current state and the action output by the policy network, resulting in an estimated state-value function. The input to the target value network is the state at the next time step, sampled from the experience replay pool, and the action output by the policy network based on that state. Its output is an estimated target state-value function, representing the maximum long-term return achievable in the next state. Furthermore, although the value network and the target value network have the same network structure, their parameters are different. The algorithm uses a delayed update approach for the target network; that is, each time the value function is updated, the algorithm uses a portion of previous experience to update the main network, and then uses the updated main network parameters to calculate the target network's value. However, the SAC algorithm does not immediately update the target network's parameters; instead, it uses a hyperparameter to control the update speed of the target network. This hyperparameter controls the update speed of the target network. Because the hyperparameter value is relatively small, the target value network's parameter update speed is slower, causing the target network's output to differ somewhat from the main network's output.
[0118] The output of the value network is the state-action value function, representing the long-term reward obtained by performing a certain action in the current state. The output of the target value network is the target state-value function, representing the maximum long-term reward obtainable in the next state.
[0119] Using a single target value network and a single value network can easily lead to overestimation. In reinforcement learning, overestimation refers to an agent overestimating the true value of an action, which may result in inaccurate or suboptimal actions taken by the agent in practical applications. This problem typically arises during the learning process because, in the initial stages, the agent has limited knowledge, and its value function needs continuous updating and optimization. If the estimate is too high, the agent may act with overconfidence, leading to performance degradation or instability.
[0120] By employing two target value networks and two value networks, and using the minimum of the two as the final estimate, the problem of overestimation can be alleviated to some extent, the variance of value estimation can be reduced, and thus the stability and efficiency of training can be improved.
[0121] Furthermore, the training steps of the model are as follows: Figure 5 As shown:
[0122] (1) Construct five neural networks using PyTorch: a policy network, two value networks, and two target value networks. Randomly initialize the policy network parameters θ and the two value network parameters φ1 and φ2. Copy the parameters from the value networks to the target value networks to obtain φ1 and φ2. targ,1 and φ targ,2 .
[0123] Initialize the experience replay pool Start a loop of K iterations, initiating a complete prediction process in each iteration, allowing the agent to make predictions at every step in the time series environment until the last time step of the sequence.
[0124] (2) For each complete prediction process, in a time series environment, after an agent starts up, the following tasks need to be completed:
[0125] Obtain the first state s1 in this transaction process;
[0126] Start a loop a specific number of times, say T times, where each iteration is a step of an agent.
[0127] (3) For each step, the following steps need to be completed:
[0128] Step 3.1: The flexible actor evaluator model obtains the current state s of the agent, and the agent obtains the action a from the policy network.
[0129] Step 3.2: The model executes action a and calculates the next state s′ reached after taking the corresponding action, the reward value r obtained, and whether the last time step d of the sequence has been reached. The transition sequence (s,a,r,s′,d) of this round is stored in the experience replay pool.
[0130] (4) Determine if the sampling count is met. If not, proceed to step 3.1 to continue interacting with the environment; otherwise, start from the experience playback pool. A specific number of round B transformation sequences {(s,a,r,s′,d)} are sampled, the target value, loss function, policy network parameter update amount, and value network parameter update amount are calculated, and the policy network, value network, and target value network are updated.
[0131] Step 4.1 Based on the target value network, calculate the target value corresponding to the sampling sequence:
[0132]
[0133] in, This represents the action acquired by the policy network in the next state s′. The entropy represents the strategy; α is a regularization coefficient that controls the degree of importance of the entropy. This indicates that the smaller of the two objective value functions is taken as the final valuation; γ is the discount factor, which is the process of discounting future returns to the present moment; r is the reward obtained by the current action.
[0134] Step 4.2: Based on the estimates of the two target value networks, calculate the mean squared error loss function of the value network. The calculation formula is shown below:
[0135]
[0136] in, |B| represents the estimated output of two target value networks based on state s and action a; |B| represents the sample size of the sampled sample transformation sequence.
[0137] Calculate the gradient corresponding to the loss function and update the two value network parameters;
[0138] Step 4.3: Use reparameterization techniques to sample actions. Calculate the loss function and gradient of the policy network, and update the policy network. The formulas for the loss function and its gradient are shown below:
[0139]
[0140] in, The loss function of the policy network is represented; This represents the gradient corresponding to the loss function; It is obtained from the policy network, rather than sampled from the experience replay pool.
[0141] Step 4.4: Update the entropy regularization coefficient α
[0142] Step 4.5: Update the parameters in the target value network using a soft update method. The update formula is shown below:
[0143] φ targ,1 =ρφ targ,1 +(1-ρ)φ1
[0144] φ targ,2 =ρφ targ,2 +(1-ρ)φ2
[0145] Where ρ is the soft update parameter; φ1 and φ2 are the parameters of the value network; φ targ,1 and φ targ,2 These are the parameters of the target value network. Using a soft update method makes the updates of the target value network smoother, preventing drastic changes and allowing for more stable network training. It also avoids the target network overfitting historical data during training, thus improving the algorithm's generalization performance and robustness.
[0146] (5) Check if the agent has reached the last state. If not, jump to (3) to continue exploring. Otherwise, determine if the entire training process has ended. If not, jump to (2) to start the next process of exploration learning. If the entire training process has ended, store the trained network model parameters.
[0147] Further, the model's effectiveness is validated and selected. The specific steps are as follows: Based on historical data, features for the test set are constructed; the trained model is then called, and the test set feature data is input. At each time step during the backtesting period, the algorithm model predicts the next time step based on the data up to the current time step, until the last time step. After the backtesting is completed, the prediction error is calculated, and the model's predictive ability is evaluated and analyzed. The model parameters are adjusted, and the backtesting results obtained with different parameters are compared. The best-performing model is selected as the final prediction model.
[0148] Furthermore, real-time data is acquired, and data preprocessing and feature engineering are performed. Based on multilayer perceptrons and combined with Fourier decomposition, trend features, seasonal features, periodic features, and holiday features are captured. Stored, trained, and selected prediction algorithm models are called, and feature-enhanced data is input to make real-time predictions of future time series.
[0149] This invention proposes a multivariate time series prediction method based on flexible actor-critic, which has the following technical advantages compared to existing technologies:
[0150] This invention constructs a multilayer perceptron and combines it with Fourier decomposition to capture trend features, seasonal features, periodic features, and holiday features. Traditional time series analysis typically uses manual construction of lag features, moving average features, difference features, sliding window features, and wavelet transform features to capture temporal correlations and periodic features in the sequence. This method is highly dependent on the knowledge and experience of domain experts, is time-consuming and complex, and struggles to remove the noise influence of historical special events, potentially leading to omissions and misjudgments. This invention introduces the construction of a multilayer perceptron, combined with Fourier decomposition, to eliminate the influence of special events and capture long-term trend terms, seasonal features, periodic features, and annual effects in the time series. It introduces features with prior knowledge of the time series data, limiting the model's fitting ability, reducing the risk of overfitting, and improving the model's robustness. By capturing the structural information of long-term series, it further improves the model's predictive performance and interpretability.
[0151] This invention introduces deep reinforcement learning to solve the time series prediction problem. Traditional time series models perform poorly in handling nonlinear and non-stationary conditions, require sophisticated parameter selection and tuning, have poor interpretability, demand high-quality data, and are prone to overfitting. This invention introduces deep reinforcement learning, which models data by constructing deep neural networks, enhancing the capture of nonlinear time series relationships; customized reward functions and policies can handle non-stationarity and heteroscedasticity; unlike traditional time series models, deep reinforcement learning can adaptively adjust parameters during training, quickly adapting to datasets with changing data distributions, thereby improving the model's predictive ability and generalization.
[0152] This invention introduces a flexible actor-critic learning approach. Reinforcement learning typically uses Monte Carlo methods to estimate gradients and optimize policies, which leads to high variance in gradient estimation and slow convergence. Direct optimization in the policy space often introduces instability and high uncertainty. This invention introduces reparameterization, reparameterizing the sampling operation of the Gaussian policy as a parameterization of the variance of the mean. By changing the mean and variance, the standard normal distribution is transformed into a sampled value that satisfies a Gaussian distribution. This achieves sampling through a differentiable transformation and can be combined with the inverse approach, improving the accuracy of gradient calculation, avoiding high variance in gradient estimation, improving the convergence speed and stability of the algorithm, and enhancing the optimization performance of the policy network.
[0153] Furthermore, this invention introduces maximum entropy reinforcement learning theory into the policy network, maximizing cumulative rewards while increasing entropy constraints. This encourages the neural network to explore various optimal possibilities in different ways, making the model more adaptable to interference and improving its robustness and generalization ability. A dual value network is used to estimate state and action values, with one value network used to optimize the policy and the other to evaluate its performance. Soft update techniques are used to update the value network parameters, mitigating errors caused by overestimation. A flexible policy update technique is introduced to alternately optimize the policy network and value network, further improving the algorithm's stability and convergence speed.
[0154] It should be noted that there is no necessary order between the above steps. Those skilled in the art will understand from the description of the embodiments of the present invention that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.
[0155] The above describes the multivariate time series prediction method based on flexible actor-critic in the embodiments of the present invention. The following describes the multivariate time series prediction system based on flexible actor-critic in the embodiments of the present invention. Please refer to [link / reference]. Figure 6 , Figure 6 This is a schematic diagram of the hardware structure of another embodiment of a flexible actor-critic multivariate time series prediction system according to an embodiment of the present invention, as shown below. Figure 6 As shown, system 10 includes: memory 101, processor 102, and a computer program stored in memory and executable on the processor. When the computer program is executed by processor 101, it performs the following steps:
[0156] Historical data is acquired, preprocessed, and then feature extraction is performed on the preprocessed historical data to complete the construction of historical features. The feature extraction operation is based on multilayer perceptron and Fourier decomposition to capture trend features, seasonal features, periodic features, and holiday features in the historical data.
[0157] A reinforcement learning element for multivariate time series prediction is defined, and a flexible actor-critic model is constructed based on the reinforcement learning element. The flexible actor-critic model includes a policy network, two value networks, and two target value networks. The flexible actor-critic model achieves soft updates and optimizations of the policy and value functions by minimizing the flexible value function and entropy regularization, as well as the importance sampling algorithm and target networks.
[0158] The flexible actor-critic model is trained based on the historical features, and after training, a multivariate time series prediction model is obtained.
[0159] The system acquires real-time data to be predicted, preprocesses the real-time data, extracts features from the preprocessed real-time data to construct real-time data features, inputs the real-time data features into the multivariate time series prediction model, obtains the output of the multivariate time series prediction model, and completes the prediction of future time series.
[0160] The specific implementation steps and method implementation examples are the same, and will not be repeated here.
[0161] Optionally, when the computer program is executed by the processor 101, it further performs the following steps:
[0162] Based on specific business processes or domain knowledge, historical data is obtained, and the historical data is the true value of the time series.
[0163] After performing data cleaning and normalization operations on the historical data, preprocessed historical data is obtained.
[0164] The specific implementation steps and method implementation examples are the same, and will not be repeated here.
[0165] Optionally, when the computer program is executed by the processor 101, it further performs the following steps:
[0166] Based on the preprocessed historical data, time-series index features, statistical index features, volatility index features, slope index features, and regression index features are calculated.
[0167] Calculate the variance inflation factor among the above-mentioned indicator features, and remove features whose variance factor value is greater than a preset threshold.
[0168] Calculate the Pearson correlation coefficient between the above-mentioned indicator features and the predicted values, and take the features with a Pearson correlation coefficient greater than the preset value as the target historical features.
[0169] The specific implementation steps and method implementation examples are the same, and will not be repeated here.
[0170] Optionally, when the computer program is executed by the processor 101, it further performs the following steps:
[0171] A nonlinear model of a multilayer perceptron is constructed, and the trend characteristics of time series are captured based on the nonlinear model of the multilayer perceptron.
[0172] A first linear model based on Fourier series is constructed, and the seasonal and periodic features of the time series are captured based on the first linear model;
[0173] A second linear model is constructed, and the holiday features of the time series are captured based on the second linear model;
[0174] By fusing the trend features, seasonal features, cyclical features, and holiday features, the feature extraction of the time series is completed, and the construction of historical features is achieved.
[0175] The specific implementation steps and method implementation examples are the same, and will not be repeated here.
[0176] Optionally, when the computer program is executed by the processor 101, it further performs the following steps:
[0177] The state elements, action elements, reward elements, and strategy elements of the multivariate time series prediction are defined.
[0178] Construct a policy network, which outputs an action policy based on the current environmental state;
[0179] Construct two value networks, which are used to combine the value obtained by performing actions in the current state with the current state and actions.
[0180] Construct two target value networks, which are used to estimate the target value of state-action pairs under the current policy.
[0181] The specific implementation steps and method implementation examples are the same, and will not be repeated here.
[0182] This invention provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example, to perform the operations described above. Figure 1 Method steps S100 to S400.
[0183] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as an external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memory in the operating environment described in the embodiments of the present invention are intended to include one or more of these and / or any other suitable types of memory.
[0184] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A multi-variate time series forecasting method based on flexible actor-critic, characterized in that, The method comprises: acquiring historical data, preprocessing the historical data, performing feature extraction on the preprocessed historical data, and completing construction of historical features, wherein the feature extraction operation captures trend features, seasonal features, periodic features, and holiday features in the historical data based on a multilayer perceptron and a Fourier decomposition method; defining reinforcement learning elements of multivariate time series prediction, constructing a flexible actor-critic model based on the reinforcement learning elements, the flexible actor-critic model comprising a policy network, two value networks, and two target value networks; wherein the flexible actor-critic model realizes soft updating and optimization of the policy and the value function by minimizing a flexible value function and entropy regularization, as well as importance sampling algorithm and target network; training the flexible actor-critic model based on the historical features, and obtaining a multivariate time series prediction model after the training is completed; acquiring real-time data to be predicted, preprocessing the real-time data, performing feature extraction on the preprocessed real-time data, completing construction of real-time data features, inputting the real-time data features into the multivariate time series prediction model, acquiring an output result of the multivariate time series prediction model, and completing prediction of future time series. 2.The multi-horizon time series forecasting method based on flexible critic-actor according to claim 1, wherein, The acquiring historical data and preprocessing the historical data comprise: acquiring historical data based on specific business processes or domain knowledge, the historical data being real values of time series; after data cleaning and normalization operations are performed on the historical data, obtaining preprocessed historical data. 3.The multi-horizon time series forecasting method based on flexible critic-actor according to claim 2, wherein, After the historical data is preprocessed, the method further comprises: calculating time series index features, statistical index features, fluctuation index features, slope index features, and regression index features based on the preprocessed historical data; calculating variance inflation factors among the various index features, and removing features with a variance factor value greater than a preset threshold value; calculating Pearson correlation coefficients between the various index features and prediction values, and taking features with a Pearson correlation coefficient greater than a preset coefficient value as target historical features.
4. The multi-horizon time series forecasting method based on flexible critic-actor according to claim 3, wherein, The performing feature extraction on the preprocessed historical data and completing construction of historical features comprise: building a nonlinear model of a multilayer perceptron, and capturing trend features of time series based on the nonlinear model of the multilayer perceptron; building a first linear model based on a Fourier series, and capturing seasonal features and periodic features of time series based on the first linear model; building a second linear model, and capturing holiday features of time series based on the second linear model; fusing the trend features, the seasonal features, the periodic features, and the holiday features to complete feature extraction of time series and construction of historical features.
5. The multi-horizon time series forecasting method based on flexible critic-actor according to claim 4, wherein, The defining reinforcement learning elements of multivariate time series prediction and constructing a flexible actor-critic model based on the reinforcement learning elements, the flexible actor-critic model comprising a policy network, two value networks, and two target value networks, comprise: Defining state elements, action elements, reward elements, policy elements of the multi-variate time series prediction; Building a policy network, which is used to output an action policy according to a current environment state; Building two value networks, which are used to combine the values obtained by performing actions in the current state according to the current state and the actions; Building two target value networks, which are used to estimate the target values of state-action pairs under the current policy.
6. A multi-horizon time series forecasting system based on flexible actor-critic, characterized in that, The system comprises a memory, a processor and a computer program stored on the memory and executable on the processor, which, when executed by the processor, implements the following steps: Obtaining historical data, preprocessing the historical data, performing feature extraction on the preprocessed historical data, and completing the construction of historical features, wherein the feature extraction operation captures trend features, seasonal features, periodic features and holiday features in the historical data based on a multi-layer perception machine and a Fourier decomposition method; Defining reinforcement learning elements of the multi-variate time series prediction, and building a flexible actor-critic model based on the reinforcement learning elements, wherein the flexible actor-critic model comprises a policy network, two value networks and two target value networks; wherein the flexible actor-critic model realizes soft update and optimization of the policy and the value function by minimizing a flexible value function and entropy regularization, as well as importance sampling algorithm and target network; Training the flexible actor-critic model based on the historical features, and obtaining a multi-variate time series prediction model after the training is completed; Obtaining real-time data to be predicted, preprocessing the real-time data, performing feature extraction on the preprocessed real-time data, completing the construction of real-time data features, inputting the real-time data features into the multi-variate time series prediction model, obtaining the output result of the multi-variate time series prediction model, and completing the prediction of future time series.
7. The flexible actor-critic based multi-horizon time series forecasting system of claim 6, wherein, The computer program, when executed by the processor, implements the following steps: Based on specific business processes or domain knowledge, obtaining historical data, which is the true value of the time series; After data cleaning and normalization operations are performed on the historical data, preprocessed historical data is obtained.
8. The flexible actor-critic based multi-horizon time series forecasting system of claim 7, wherein, The computer program, when executed by the processor, further implements the following steps: Based on the preprocessed historical data, calculating time series index features, statistical index features, volatility index features, slope index features and regression index features; Calculating the variance inflation factor among the above-mentioned various index features, and eliminating features with a variance factor value greater than a preset threshold value; Calculating the Pearson correlation coefficient between the above-mentioned various index features and the predicted value, and taking features with a Pearson correlation coefficient greater than a preset coefficient value as target historical features.
9. The flexible actor-critic based multi-horizon time series forecasting system of claim 8, wherein, The computer program, when executed by the processor, further implements the following steps: Building a non-linear model of a multi-layer perception machine, and capturing trend features of the time series based on the non-linear model of the multi-layer perception machine; Building a first linear model based on Fourier series, and capturing seasonal features and periodic features of the time series based on the first linear model; A second linear model is built, and holiday features of the time series are captured based on the second linear model; The trend features, the seasonal features, the periodic features and the holiday features are fused to complete feature extraction of the time series and construction of the historical features.
10. A non-transitory computer readable storage medium, comprising: The non-volatile computer readable storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to make the one or more processors execute the flexible actor-critic based multi-element time series prediction method in any one of claims 1-5.