Multivariate time series prediction method and system based on near-end strategy optimization

Abstract

The invention discloses a multivariate time series prediction method and system based on near-end strategy optimization, and the method comprises the steps: obtaining historical data, carrying out the preprocessing of the historical data, carrying out the feature extraction operation, and completing the construction of historical features; defining reinforcement learning elements of multivariate time series prediction, and constructing a near-end strategy optimization model based on the reinforcement learning elements; training the near-end strategy optimization model based on historical features, and obtaining a multivariate time series prediction model after training is completed; real-time data to be predicted are obtained, the real-time data are preprocessed, feature extraction operation is carried out, construction of real-time data features is completed, the real-time data features are input into the multivariate time series prediction model, an output result is obtained, and prediction of a future time series is completed. According to the embodiment of the invention, efficient learning and optimization can be carried out in a large-scale environment under limited computing resources, the problem of overfitting can be avoided, and the stability, robustness and generalization ability of the model can be greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent agents, and in particular to a multi-element time series prediction method and system based on proximal policy optimization. BACKGROUND

[0002] With the rapid advancement of the information revolution and the rapid development of technology, the thinking, innovation and application of science and technology are also changing the original competitive pattern and ecological system of many industries such as the Internet and finance. Under this new situation, the Internet, big data and artificial intelligence technologies that are accelerating development are reshaping the patterns and paths of financial development. The role, status and development of time series prediction are becoming increasingly important in various fields.

[0003] Traditional time series prediction methods include moving average, exponential smoothing, ARIMA model, etc. These methods can predict future trends by fitting historical data, have certain accuracy and interpretability, but often require stationary data tests, and have poor processing effect on non-linear, non-stationary, etc. The ability to capture category features and high-dimensional data features is limited, and it is difficult to handle multi-dimensional time series prediction. It is easily affected by outliers and has poor model generalization ability.

[0004] Machine learning methods include regression models, support vector machines, recurrent neural networks, long short-term memory networks, etc. Although they can handle more complex data and do not require much preprocessing, they require more computing resources and time for training, and have high requirements for model parameter selection and adjustment. The model has poor interpretability, high requirements for data quality, and is prone to overfitting.

[0005] Therefore, the prior art still needs to be further developed. SUMMARY

[0006] To solve the above technical problems, the embodiments of the present application provide a multi-element time series prediction method and system based on proximal policy optimization, which can solve the technical problems that the machine learning algorithm in the prior art requires more computing resources and time for training, has high requirements for model parameter selection and adjustment, has poor model interpretability, has high requirements for data quality, and is prone to overfitting when predicting time series.

[0007] The first aspect of the embodiments of the present application provides a multi-element time series prediction method based on proximal policy optimization, the method comprising:

[0008] Obtaining historical data, preprocessing the historical data, performing feature extraction operation on the preprocessed historical data, and completing the construction of historical features, wherein the feature extraction operation captures category features based on word embedding and automatically learns cross-features of time series based on deep learning and factorization machines.

[0009] A reinforcement learning element for multivariate time series prediction is defined, and a proximal policy optimization model is constructed based on the reinforcement learning element. The proximal policy optimization model includes a policy network and a value function network. The loss function of the policy network adopts a truncated advantage function, and the policy network uses multiple mini-batch samples for policy updates.

[0010] The near-end strategy optimization model is trained based on the historical features. After training, a multivariate time series prediction model is obtained.

[0011] The system acquires real-time data to be predicted, preprocesses the real-time data, extracts features from the preprocessed real-time data to construct real-time data features, inputs the real-time data features into the multivariate time series prediction model, obtains the output of the multivariate time series prediction model, and completes the prediction of future time series.

[0012] Optionally, the step of acquiring historical data and preprocessing the historical data includes:

[0013] Based on specific business processes or domain knowledge, historical data is obtained, and the historical data is the true value of the time series.

[0014] After performing data cleaning and normalization operations on the historical data, preprocessed historical data is obtained.

[0015] Optionally, after performing data cleaning and normalization operations on the historical data to obtain preprocessed historical data, the method further includes:

[0016] Based on the preprocessed historical data, time-series index features, statistical index features, volatility index features, slope index features, and regression index features are calculated.

[0017] Calculate the variance inflation factor among the above-mentioned indicator features, and remove features whose variance factor value is greater than a preset threshold.

[0018] Calculate the Pearson correlation coefficient between the above-mentioned indicator features and the predicted values, and take the features with a Pearson correlation coefficient greater than the preset value as the target historical features.

[0019] Optionally, feature extraction is performed on the preprocessed historical data to construct historical features, including:

[0020] A factorization machine and a deep neural network are pre-constructed, wherein the deep neural network and the factorization machine share the same embedding layer;

[0021] The categorical features in the target historical features are obtained, the categorical features are converted into data codes, the data codes are input into the embedding layer, and the output of the embedding layer is obtained to complete the capture of categorical features. The embedding layer is used to map high-dimensional feature vectors into low-dimensional real number vectors and to convert sparse categorical features into dense embedding features.

[0022] The target historical features are divided into different feature domains. The data encoding and the output of the embedding layer are input into a deep neural network to obtain a first output value. The feature data of different feature domains are input into a factorization machine to obtain a second output value. The cross features of the time series are obtained based on the first output value and the second output value. Each feature domain corresponds to a different sparse embedding layer and a dense embedding layer.

[0023] Based on the captured categorical features and the cross features of the time series, the historical features are constructed.

[0024] Optionally, reinforcement learning elements for multivariate time series prediction are defined, and a proximal policy optimization model is constructed based on these elements. The proximal policy optimization model includes a policy network and a value function network, comprising:

[0025] The state elements, action elements, reward elements, and strategy elements of the multivariate time series prediction are defined.

[0026] Construct a policy network, which takes the environment state as input and outputs the probability distribution of all action elements under the current policy element;

[0027] Construct a value function network, which is used to take the environmental state as input and output an estimated value of the current environmental state;

[0028] The policy network and the value function network constitute the proximal policy optimization model.

[0029] A second aspect of this invention provides a multivariate time series forecasting system based on proximal strategy optimization. The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program performs the following steps:

[0030] Historical data is acquired, preprocessed, and feature extraction is performed on the preprocessed historical data to complete the construction of historical features. In the feature extraction operation, categorical features are captured based on word embedding, and cross features of time series are automatically learned based on deep learning and factorization machine.

[0031] A reinforcement learning element for multivariate time series prediction is defined, and a proximal policy optimization model is constructed based on the reinforcement learning element. The proximal policy optimization model includes a policy network and a value function network. The loss function of the policy network adopts a truncated advantage function, and the policy network uses multiple mini-batch samples for policy updates.

[0032] The near-end strategy optimization model is trained based on the historical features. After training, a multivariate time series prediction model is obtained.

[0033] The system acquires real-time data to be predicted, preprocesses the real-time data, extracts features from the preprocessed real-time data to construct real-time data features, inputs the real-time data features into the multivariate time series prediction model, obtains the output of the multivariate time series prediction model, and completes the prediction of future time series.

[0034] Optionally, when the computer program is executed by the processor, it performs the following steps:

[0035] Based on specific business processes or domain knowledge, historical data is obtained, and the historical data is the true value of the time series.

[0036] After performing data cleaning and normalization operations on the historical data, preprocessed historical data is obtained.

[0037] Optionally, when the computer program is executed by the processor, it further performs the following steps:

[0038] Based on the preprocessed historical data, time-series index features, statistical index features, volatility index features, slope index features, and regression index features are calculated.

[0039] Calculate the variance inflation factor among the above-mentioned indicator features, and remove features whose variance factor value is greater than a preset threshold.

[0040] Calculate the Pearson correlation coefficient between the above-mentioned indicator features and the predicted values, and take the features with a Pearson correlation coefficient greater than the preset value as the target historical features.

[0041] Optionally, when the computer program is executed by the processor, it further performs the following steps:

[0042] A factorization machine and a deep neural network are pre-constructed, wherein the deep neural network and the factorization machine share the same embedding layer;

[0043] The categorical features in the target historical features are obtained, the categorical features are converted into data codes, the data codes are input into the embedding layer, and the output of the embedding layer is obtained to complete the capture of categorical features. The embedding layer is used to map high-dimensional feature vectors into low-dimensional real number vectors and to convert sparse categorical features into dense embedding features.

[0044] The target historical features are divided into different feature domains. The data encoding and the output of the embedding layer are input into a deep neural network to obtain a first output value. The feature data of different feature domains are input into a factorization machine to obtain a second output value. The cross features of the time series are obtained based on the first output value and the second output value. Each feature domain corresponds to a different sparse embedding layer and a dense embedding layer.

[0045] Based on the captured categorical features and the cross features of the time series, the historical features are constructed.

[0046] A third aspect of the present invention provides a non-volatile computer-readable storage medium, characterized in that the non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to perform the above-described multivariate time series prediction method based on proximal policy optimization.

[0047] The technical solution provided in this invention involves acquiring historical data, preprocessing the historical data, and performing feature extraction to construct historical features; defining reinforcement learning elements for multivariate time series prediction, and constructing a proximal policy optimization model based on these elements; training the proximal policy optimization model based on historical features to obtain a multivariate time series prediction model; acquiring real-time data to be predicted, preprocessing the real-time data, and performing feature extraction to construct real-time data features; inputting the real-time data features into the multivariate time series prediction model to obtain the output results and complete the prediction of future time series. This invention enables efficient learning and optimization in large-scale environments with limited computing resources, avoids overfitting problems, and greatly improves the model's stability, robustness, and generalization ability, thereby increasing the efficiency of multivariate time series prediction. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating an embodiment of a multivariate time series prediction method based on proximal strategy optimization according to the present invention.

[0049] Figure 2 This is a schematic diagram of the structure of a proximal policy optimization model in an embodiment of a multivariate time series prediction method based on proximal policy optimization according to an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the network structure of the policy network in an embodiment of a multivariate time series prediction method based on near-end policy optimization according to an embodiment of the present invention.

[0051] Figure 4 This is a flowchart of the network training process of a proximal policy optimization model, which is an embodiment of a multivariate time series prediction method based on proximal policy optimization, according to an embodiment of the present invention.

[0052] Figure 5 This is a schematic diagram of the multivariate time series prediction process of a multivariate time series prediction model according to an embodiment of a multivariate time series prediction method based on near-end strategy optimization in this invention.

[0053] Figure 6 This is a schematic diagram of the hardware structure of another embodiment of a multivariate time series prediction system based on proximal strategy optimization in this invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0056] Please see Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of a multivariate time series prediction method based on proximal strategy optimization according to an embodiment of the present invention. Figure 1 As shown, it includes:

[0057] Step S100: Obtain historical data, preprocess the historical data, perform feature extraction on the preprocessed historical data, and complete the construction of historical features. In the feature extraction operation, categorical features are captured based on word embedding, and cross features of time series are automatically learned based on deep learning and factorization machine.

[0058] Step S200: Define reinforcement learning elements for multivariate time series prediction, and construct a proximal policy optimization model based on the reinforcement learning elements. The proximal policy optimization model includes a policy network and a value function network. The loss function of the policy network adopts a truncated advantage function, and the policy network uses multiple mini-batch samples for policy updates.

[0059] Step S300: Train the near-end strategy optimization model based on the historical features. After training, a multivariate time series prediction model is obtained.

[0060] Step S400: Obtain the real-time data to be predicted, preprocess the real-time data, perform feature extraction on the preprocessed real-time data to complete the construction of real-time data features, input the real-time data features into the multivariate time series prediction model, obtain the output results of the multivariate time series prediction model, and complete the prediction of future time series.

[0061] In specific implementation, this embodiment of the invention acquires historical data, performs preprocessing operations such as anomaly detection, missing value imputation, and normalization, captures categorical features through word embedding, and automatically learns the cross features of time series using deep learning and factorization machines to complete the construction of feature engineering.

[0062] Define the reinforcement learning elements for multivariate time series prediction; adopt a proximal policy optimization model to build a policy network and a value network; use a truncated advantage function to calculate the policy loss function to limit the policy update magnitude; sample multiple mini-batch samples for policy updates to avoid excessive variance caused by large batch updates.

[0063] Based on a preset ratio, the historical features are divided into a training set, a test set, and a validation set. The proximal policy optimization model is trained using the training set, and its effectiveness is verified using the validation set. Model parameters are adjusted based on the verification results. The adjusted proximal policy optimization model is then tested using the test set. When the test results meet the criteria, the proximal policy optimization model is selected, resulting in a multivariate time series prediction model. For example: based on historical data, maintaining the sequential nature of the sequence, the training set, test set, and validation set are divided in a 6:3:1 ratio. Based on the data in the training set, model parameters are updated by comparing the differences between the current policy and historical policies, completing model training. The trained model is saved, backtested, and its effectiveness verified. Model parameters are then adjusted, completing model selection.

[0064] The selected time series forecasting model is deployed to the production environment to make real-time predictions of future time series.

[0065] This invention employs historical data acquisition and preprocessing; word embedding to capture categorical features; deep learning and factorization machines to automatically learn the cross features of time series; a proximal policy optimization algorithm to build a policy network and a value network, introducing a truncation mechanism; limiting the step size of policy updates during iteration by using an importance sampling ratio; and using multiple mini-batch samples for policy updates to train the model. Backtesting is then performed based on the trained model to verify its effectiveness, and models with strong fitting and generalization abilities are selected for real-time prediction of future time series. This invention achieves efficient learning and optimization in large-scale environments with limited computing resources, avoids overfitting problems, and significantly improves the model's stability, robustness, and generalization ability.

[0066] Furthermore, historical data is acquired, and the historical data is preprocessed, including:

[0067] Based on specific business processes or domain knowledge, historical data is obtained, and the historical data is the true value of the time series.

[0068] After performing data cleaning and normalization operations on the historical data, preprocessed historical data is obtained.

[0069] In practice, the true value of the time series is obtained based on specific business processes or domain knowledge.

[0070] Data cleaning operations include: deleting duplicate values ​​from the original data summary; filling missing values ​​with lag 7, lag 14 data, and linear interpolation; defining values ​​smaller than a specific value and negative values ​​as outliers and processing them with piecewise linear smoothing; and performing logarithmic processing on the data to remove heteroscedasticity from the sequence, reduce the impact of outliers, and increase the stationarity of the data.

[0071] The lag feature in time series refers to using data from a period of time preceding a given point in time as a feature to predict the value at that point in time. Lag features are typically widely used in time series forecasting because they are intuitive and interpretable, and can capture trends and seasonality in time series data.

[0072] In time series analysis, lag features include lag order 1, lag order 2, lag order 3, ..., lag order n, where n represents the number of time steps in the lag. For example, a lag order 7 feature means using data from the previous 7 time steps as features to predict the value at the current time step.

[0073] Lag features can capture the correlation and trend of time series. In this invention, lag 7 and lag 14 features are selected for filling in order to preserve the periodic effect of the time series, thereby obtaining the best prediction effect.

[0074] A detailed description of the linear interpolation fill method is shown below:

[0075] Linear interpolation fills missing values ​​with the linear average of the two known values ​​preceding and following them. Let y be the i-th data point in the sequence. i Let k be the number of data points between the i-th and k-th data points. Then, the number of data points requiring interpolation during this period is kji. For these data points requiring interpolation, we can use the known data points (j-1, y) before and after them. j-1 (k+1, y) and (k+1, y) k+1 Calculate its interpolation y i :

[0076]

[0077] Among them, (t) k+1 -y j-1 ) represents the difference between the known data points before and after the input, and k-j+2 represents the input data points to be interpolated plus the known data points at both ends. Using the above method, linear interpolation can replace missing values ​​with the linear average of the known values ​​before and after them.

[0078] In regression analysis, outliers can easily cause the regression curve to deviate from the true trend, distorting the true relationship between variables and weakening model performance. Therefore, outlier processing is necessary. Winsorization is a commonly used data processing method that can reduce the impact of extreme values ​​(outliers). The basic idea of ​​winsorization is to replace values ​​less than the p-th percentile (p% quantile) and greater than the q-th percentile (q% quantile) with their respective p-th and q-th percentile values. "Winsor 1%" in winsorization refers to replacing values ​​less than the 1st percentile and greater than the 99th percentile with their respective 1% and 99th percentile values, essentially retaining 99% of the data and removing the smallest and largest 1%. This method is often used to handle data with a large number of extreme outliers, effectively reducing their impact and improving data stability and robustness. The specific value of this invention is 1%, based on Winsor's 1% tail reduction processing. Considering practical application scenarios such as peak holidays and other special time periods, the maximum value generated is also important information that needs to be predicted in time series. Therefore, only the value less than 1% is processed.

[0079] Normalization processing includes: normalizing time series data to obtain preprocessed historical data, thereby reducing the impact of data with different dimensions on feature weights and improving the convergence speed and stability of the model.

[0080] Furthermore, after performing data cleaning and normalization operations on the historical data to obtain preprocessed historical data, the process further includes:

[0081] Based on the preprocessed historical data, time-series index features, statistical index features, volatility index features, slope index features, and regression index features are calculated.

[0082] Calculate the variance inflation factor among the above-mentioned indicator features, and remove features whose variance factor value is greater than a preset threshold.

[0083] Calculate the Pearson correlation coefficient between the above-mentioned indicator features and the predicted values, and take the features with a Pearson correlation coefficient greater than the preset value as the target historical features.

[0084] In practice, time-series indicators, statistical indicators, volatility indicators, slope indicators, and regression indicators are calculated based on historical data. The variance inflation factor is used to detect multicollinearity among features, and features with high variance inflation factor values ​​are eliminated to reduce the impact of collinearity. The Pearson correlation coefficient between features and predicted values ​​is calculated as a standard to measure the effectiveness of features. The effectiveness of features is tested, and features with correlation values ​​greater than 0.1 are selected to participate in model training.

[0085] The above-mentioned indicators can be collectively referred to as characteristic variables, which are mainly classified based on the nature and purpose of the characteristics.

[0086] Time-series indicators: These indicators are generally calculated based on time-series data and include, but are not limited to, the trends and seasonality of the time series. For example, the mean, variance, and autocorrelation of a time series are time-series indicators.

[0087] Statistical indicators: These indicators are generally derived from statistical analysis of time series data, including but not limited to percentiles, skewness, and kurtosis. These indicators can reflect the distribution characteristics of the time series.

[0088] Volatility indicators: These indicators are generally used to describe the volatility of time series data, including but not limited to volatility, standard deviation, and coefficient of variation. These indicators can be used to measure the risk level of time series data.

[0089] Slope-related indicators: These indicators generally describe the trend characteristics of time series, including but not limited to slope, trend line intercepts, etc. These indicators can be used to predict the future trend of time series.

[0090] Regression indicators: These indicators are generally used to describe the relationship between time series data and other variables, such as the relationship between time series data and macroeconomic indicators, meteorological data, etc. Common regression indicators include correlation coefficients, regression coefficients, and residuals.

[0091] Furthermore, feature extraction is performed on the preprocessed historical data to construct historical features, including:

[0092] A factorization machine and a deep neural network are pre-constructed, wherein the deep neural network and the factorization machine share the same embedding layer;

[0093] The categorical features in the target historical features are obtained, the categorical features are converted into data codes, the data codes are input into the embedding layer, and the output of the embedding layer is obtained to complete the capture of categorical features. The embedding layer is used to map high-dimensional feature vectors into low-dimensional real number vectors and to convert sparse categorical features into dense embedding features.

[0094] The target historical features are divided into different feature domains. The data encoding and the output of the embedding layer are input into a deep neural network to obtain a first output value. The feature data of different feature domains are input into a factorization machine to obtain a second output value. The cross features of the time series are obtained based on the first output value and the second output value. Each feature domain corresponds to a different sparse embedding layer and a dense embedding layer.

[0095] Based on the captured categorical features and the cross features of the time series, the historical features are constructed.

[0096] In practice, for categorical features, label encoding is first used to convert them into data encoding. Then, the numerical encoding is used as input, and the high-dimensional feature vector is mapped to a low-dimensional real number vector through the embedding layer. This transforms sparse categorical features into dense embedding features, which can reduce the number of model parameters, improve computational efficiency, and better capture the correlation between features.

[0097] A combined deep learning and factorization machine model is constructed: features are divided into different feature domains, each corresponding to different sparse and dense embedding layers. The dimension of each feature domain's embedding layer is consistent with the dimension of the latent vector of the factorization machine. A factorization machine and a deep neural network are constructed, both sharing the same embedding layer. The factorization machine performs pairwise cross-interactions on the embedding vectors of different feature domains to obtain the final cross-feature representation. The deep neural network first connects the input dense features and the sparse features encoded by the embedding layer. After processing through several fully connected layers and activation functions, a single output value is obtained. This value, along with the output of the factorization machine, is input into the final output layer of the model to participate in the final target fitting.

[0098] Furthermore, reinforcement learning elements for multivariate time series prediction are defined, and a proximal policy optimization model is constructed based on these elements. The proximal policy optimization model includes a policy network and a value function network, comprising:

[0099] The state elements, action elements, reward elements, and strategy elements of the multivariate time series prediction are defined.

[0100] Construct a policy network, which takes the environment state as input and outputs the probability distribution of all action elements under the current policy element;

[0101] Construct a value function network, which is used to take the environmental state as input and output an estimated value of the current environmental state;

[0102] The policy network and the value function network constitute the proximal policy optimization model.

[0103] In practice, reinforcement learning elements include, but are not limited to, the definitions of reinforcement learning elements such as state, action, and reward.

[0104] Definition of elements in temporal prediction reinforcement learning: such as Figure 2 As shown, in reinforcement learning, an agent learns the optimal policy by interacting with its environment. In time series prediction, the environment refers to the current time series data and its associated features. At each time step, the agent selects an action, receives a reward, and then moves to the next time step, selecting another action in the new state. The agent's goal is to minimize prediction error and improve prediction accuracy by learning the optimal policy.

[0105] Time series prediction based on reinforcement learning includes the following elements:

[0106] State: All information observed by the agent at a specific point in time, including current historical observations and time series indicators such as time series indicators, statistical indicators, and volatility indicators, including seasonality, trend, periodicity, and other external variables related to the prediction target, used to predict the value at the next point in time.

[0107] Action: An agent's action is to output a definite value in a continuous action space based on the current state and the agent's policy. The definite value designed in this invention is a non-negative predicted value.

[0108] Reward: After the agent performs an action, the accuracy of the prediction is used as the reward or penalty for that action and is fed back to the agent.

[0109]

[0110] Where actual_value represents the actual value and predict_value represents the predicted value.

[0111] Policy: Specifies the action to be taken in each state to maximize long-term gains. It can be a deterministic or stochastic policy, and is typically represented as a function that takes states as input and outputs actions.

[0112] Based on the current state and reward, the reinforcement learning algorithm selects an action, outputs a predicted value, and updates the policy to improve the accuracy and stability of predictions. Through iterative learning, the algorithm can learn how to take the optimal predictive action in different states, thereby improving the accuracy and stability of time series predictions.

[0113] In reinforcement learning, an intelligent agent differs from intelligent robots and smartphones. An intelligent agent is an entity capable of observing its environment, making decisions, and learning through interaction with it. The agent's goal is to learn optimal behavioral strategies through interaction with the environment, maximizing its rewards. The agent can perceive the state of the environment and choose an action based on that state. The environment provides feedback on the agent's actions, evaluating them through a reward function. The agent adjusts its strategy based on the reward feedback to better adapt to the environment.

[0114] Proximal Policy Optimization Model: Reinforcement learning involves an agent continuously interacting with its environment to perceive it, and the resulting optimal policy parameters are stored in the network. This invention constructs two identical networks: a policy network and a value function network. The policy network takes the environment state as input and outputs the probability distribution of all actions under the current policy; the value function network takes the environment state as input and outputs an estimated value of the current state, used to calculate the advantage function and update the policy.

[0115] The network structures of policy networks and value networks, such as Figure 3 As shown, the network can be divided into three main parts: multiple convolutional layers and activation layers, fully connected layers, and linear activation layers. Convolutional layers can extract local features of the state at different spatial locations. Activation layers can introduce nonlinearity, effectively enhancing the network's representational ability, accelerating the training process, and avoiding the gradient vanishing problem. Using multiple convolutional layers and ReLU activation layers can further increase the network's depth and complexity, thereby improving the network's ability to abstractly represent and generalize states, helping to reduce dependence on large amounts of data, and improving learning efficiency and performance. Fully connected layers can supplement the model's learning based on the features extracted from the convolutional layers, thus better adapting to the characteristics of the input data and improving the performance of the policy and value functions.

[0116] Furthermore, the training steps of the model are as follows: Figure 4 As shown:

[0117] (1) Based on the network structure, a policy network and a value function network are constructed using PyTorch, and the policy parameters θ_0 and value function parameters φ_0 are randomly initialized. An experience replay pool is initialized, and a loop of K iterations is started. In each loop, a complete prediction process is initiated, allowing the agent to make predictions at each step in the time series environment until the last time step of the sequence.

[0118] (2) For each complete prediction process, in a time series environment, after an agent starts up, the following tasks need to be completed:

[0119] Step 2.1: Based on the current policy of the policy network, collect a certain amount of empirical data through interaction with the environment. Each piece of empirical data includes: current state s t Execute action a t The rewards received Next state s t+1 And whether the last time step d of the sequence has been reached. t .

[0120] Step 2.2: For each piece of empirical data Estimate the current state value V(s) using the current value function network. t ), calculate the advantage function estimate

[0121] Advantage function estimation Indicates that in state s t Take action a t The formula for calculating the relative merits of the current strategy is as follows:

[0122]

[0123] Where T represents the maximum time step of the current complete prediction process; γ represents the discount factor; c j This represents the truncation function used in the near-end policy optimization algorithm; s represents the reward signal at time step i; i V(s) represents the state at time step i; i ) indicates that in state s i The state value function obtained by using the current value function V.

[0124] Step 2.3: Calculate the importance sampling ratio, which is the ratio between the new strategy and the old strategy, to determine the magnitude of the update. The calculation formula is shown below:

[0125]

[0126] Where, π θ (at |s t ) represents the state given the current state s t In this case, the new strategy π is based on the parameter θ for action a t The probability distribution; This indicates that given the current state s t In this case, the old strategy π is based on the parameter θ for action a. t The probability distribution.

[0127] Step 2.4: Based on the calculated advantage estimate and importance sampling ratio, calculate the objective function required to update the strategy. The formula for the objective function is shown below:

[0128]

[0129] in, Indicates will It is restricted to [1-∈, 1+∈]; ∈ is a hyperparameter that indicates the range of the cutoff.

[0130] Calculate the gradient of the objective function, sample the stochastic gradient descent method with an adaptive learning rate, and update the policy parameters θ to maximize the objective function L(θ). The updated parameters of the policy parameters θ are shown below:

[0131]

[0132]

[0133] Step 2.5: Update the value function parameter φ k+1 This minimizes the mean error of the value function.

[0134]

[0135] in, Let V represent a set of empirical data under the current strategy, φ represent the parameters of the value function, k represent the number of iterations, and V represent the value of the data. φ (s t ) represents the value function with respect to state s t The estimated value. By minimizing the mean squared error of the value function, the value function can more accurately estimate the discounted return, thereby improving the learning efficiency of the policy.

[0136] Step 3: Repeat step 2 until the model converges or reaches the specified number of training rounds, then store the trained network model and the learned parameters.

[0137] Further, the model's effectiveness is validated and selected. The specific steps are as follows: Based on historical data, features for the test set are constructed; the trained model is then called, and the test set feature data is input. At each time step during the backtesting period, the algorithm model predicts the next time step based on the data up to the current time step, until the last time step. After the backtesting is completed, the prediction error is calculated, and the model's predictive ability is evaluated and analyzed. The model parameters are adjusted, and the backtesting results obtained with different parameters are compared. The best-performing model is selected as the final prediction model.

[0138] Furthermore, such as Figure 5 As shown, real-time data is acquired, and data preprocessing and feature engineering are performed. Word embedding is used to capture categorical features, and deep learning and factorization machine are used to automatically learn the cross features of the model. The stored trained and selected prediction algorithm model is called, the feature data is input, and the future time series is predicted in real time. The prediction results are post-processed, such as inverse normalization, and the processed prediction results are used for practical applications.

[0139] This invention proposes a multivariate time series forecasting method based on proximal strategy optimization, which, compared to existing technologies,

[0140] This invention introduces deep learning and factorization machines to automatically learn the cross features of time series data. Traditional time series models typically require manual feature extraction and correlation analysis, which limits their ability to mine high-dimensional features. This invention introduces deep neural networks and factorization machines to automatically extract features and learn high-order cross features to capture complex interaction patterns, improving the model's adaptability to nonlinear datasets and enhancing its predictive performance. It also simplifies the model construction process, improving its scalability and generality; and enhances the model's ability to process various time series data, including datasets with different time scales, frequencies, and lengths, significantly improving its generalization ability.

[0141] This invention introduces word embeddings to capture categorical features. Traditional time series analysis typically encodes categorical features using one-hot encoding, leading to sparse feature vectors. The introduction of numerous dimensions not only risks the curse of dimensionality but also makes model training extremely slow, or even impossible. This invention uses word embeddings to map categorical features into a low-dimensional space, reducing feature dimensionality. By learning the relationships between features, the model's generalization ability can be improved. Furthermore, word embeddings can initialize model parameters through pre-training, accelerating convergence and improving model performance.

[0142] This invention introduces a proximal policy optimization model. In traditional reinforcement learning methods, model updates may degrade policy performance, leading to unstable model training and even difficulty in convergence. While relative policy optimization algorithms can improve model stability and performance, their complex second-order methods result in high computational costs and long training times, making them unsuitable for large-scale applications.

[0143] This invention introduces a proximal policy optimization model using a first-order method to improve the stability of the training process and model performance under limited computational resources. It also introduces a gradient truncation mechanism, limiting the step size of policy updates during iterations by controlling the importance sampling ratio, thus avoiding excessive update amplitudes and preventing abrupt changes during optimization, thereby improving the algorithm's stability and convergence. Furthermore, it employs multiple mini-batch samples for policy updates to avoid biases that might result from updating from a single sample. By utilizing all sampled mini-batch samples, it fully leverages existing data, avoids redundant sampling, and improves sampling efficiency. Multiple mini-batch samples can come from different environments or trajectories, resulting in more comprehensive policy updates that adapt to different environments, thereby improving the algorithm's robustness and generalization ability.

[0144] It should be noted that there is no necessary order between the above steps. Those skilled in the art will understand from the description of the embodiments of the present invention that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0145] The multivariate time series prediction method based on proximal policy optimization in the embodiments of the present invention has been described above. The multivariate time series prediction system based on proximal policy optimization in the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 6 , Figure 6 This is a schematic diagram of the hardware structure of another embodiment of a multivariate time series prediction system based on proximal strategy optimization, as described in this invention. Figure 6 As shown, system 10 includes: memory 101, processor 102, and a computer program stored in memory and executable on the processor. When the computer program is executed by processor 101, it performs the following steps:

[0146] Historical data is acquired, preprocessed, and feature extraction is performed on the preprocessed historical data to complete the construction of historical features. In the feature extraction operation, categorical features are captured based on word embedding, and cross features of time series are automatically learned based on deep learning and factorization machine.

[0147] A reinforcement learning element for multivariate time series prediction is defined, and a proximal policy optimization model is constructed based on the reinforcement learning element. The proximal policy optimization model includes a policy network and a value function network. The loss function of the policy network adopts a truncated advantage function, and the policy network uses multiple mini-batch samples for policy updates.

[0148] The near-end strategy optimization model is trained based on the historical features. After training, a multivariate time series prediction model is obtained.

[0149] The system acquires real-time data to be predicted, preprocesses the real-time data, extracts features from the preprocessed real-time data to construct real-time data features, inputs the real-time data features into the multivariate time series prediction model, obtains the output of the multivariate time series prediction model, and completes the prediction of future time series.

[0150] The specific implementation steps and method implementation examples are the same, and will not be repeated here.

[0151] Optionally, when the computer program is executed by the processor 101, it further performs the following steps:

[0152] Based on specific business processes or domain knowledge, historical data is obtained, and the historical data is the true value of the time series.

[0153] After performing data cleaning and normalization operations on the historical data, preprocessed historical data is obtained.

[0154] The specific implementation steps and method implementation examples are the same, and will not be repeated here.

[0155] Optionally, when the computer program is executed by the processor 101, it further performs the following steps:

[0156] Based on the preprocessed historical data, time-series index features, statistical index features, volatility index features, slope index features, and regression index features are calculated.

[0157] Calculate the variance inflation factor among the above-mentioned indicator features, and remove features whose variance factor value is greater than a preset threshold.

[0158] Calculate the Pearson correlation coefficient between the above-mentioned indicator features and the predicted values, and take the features with a Pearson correlation coefficient greater than the preset value as the target historical features.

[0159] The specific implementation steps and method implementation examples are the same, and will not be repeated here.

[0160] Optionally, when the computer program is executed by the processor 101, it further performs the following steps:

[0161] A factorization machine and a deep neural network are pre-constructed, wherein the deep neural network and the factorization machine share the same embedding layer;

[0162] The categorical features in the target historical features are obtained, the categorical features are converted into data codes, the data codes are input into the embedding layer, and the output of the embedding layer is obtained to complete the capture of categorical features. The embedding layer is used to map high-dimensional feature vectors into low-dimensional real number vectors and to convert sparse categorical features into dense embedding features.

[0163] The target historical features are divided into different feature domains. The data encoding and the output of the embedding layer are input into a deep neural network to obtain a first output value. The feature data of different feature domains are input into a factorization machine to obtain a second output value. The cross features of the time series are obtained based on the first output value and the second output value. Each feature domain corresponds to a different sparse embedding layer and a dense embedding layer.

[0164] Based on the captured categorical features and the cross features of the time series, the historical features are constructed.

[0165] The specific implementation steps and method implementation examples are the same, and will not be repeated here.

[0166] Optionally, when the computer program is executed by the processor 101, it further performs the following steps:

[0167] The state elements, action elements, reward elements, and strategy elements of the multivariate time series prediction are defined.

[0168] Construct a policy network, which takes the environment state as input and outputs the probability distribution of all action elements under the current policy element;

[0169] Construct a value function network, which is used to take the environmental state as input and output an estimated value of the current environmental state;

[0170] The policy network and the value function network constitute the proximal policy optimization model.

[0171] The specific implementation steps and method implementation examples are the same, and will not be repeated here.

[0172] This invention provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example, to perform the operations described above. Figure 1 Method steps S100 to S400.

[0173] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as an external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memory in the operating environment described in the embodiments of the present invention are intended to include one or more of these and / or any other suitable types of memory.

[0174] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multivariate time series forecasting method based on proximal strategy optimization, characterized in that, The methods include: Historical data is acquired, preprocessed, and feature extraction is performed on the preprocessed historical data to complete the construction of historical features. In the feature extraction operation, categorical features are captured based on word embedding, and cross features of time series are automatically learned based on deep learning and factorization machine. A reinforcement learning element for multivariate time series prediction is defined, and a proximal policy optimization model is constructed based on the reinforcement learning element. The proximal policy optimization model includes a policy network and a value function network. The loss function of the policy network adopts a truncated advantage function, and the policy network uses multiple mini-batch samples for policy updates. The near-end strategy optimization model is trained based on the historical features. After training, a multivariate time series prediction model is obtained. The system acquires real-time data to be predicted, preprocesses the real-time data, extracts features from the preprocessed real-time data to construct real-time data features, inputs the real-time data features into the multivariate time series prediction model, obtains the output of the multivariate time series prediction model, and completes the prediction of future time series.

2. The multivariate time series prediction method based on proximal strategy optimization according to claim 1, characterized in that, The acquisition of historical data and the preprocessing of the historical data include: Based on specific business processes or domain knowledge, historical data is obtained, and the historical data is the true value of the time series. After performing data cleaning and normalization operations on the historical data, preprocessed historical data is obtained.

3. The multivariate time series prediction method based on proximal strategy optimization according to claim 2, characterized in that, After performing data cleaning and normalization operations on the historical data to obtain preprocessed historical data, the process further includes: Based on the preprocessed historical data, time-series index features, statistical index features, volatility index features, slope index features, and regression index features are calculated. Calculate the variance inflation factor among the above-mentioned indicator features, and remove features whose variance factor value is greater than a preset threshold. Calculate the Pearson correlation coefficient between the above-mentioned indicator features and the predicted values, and take the features with a Pearson correlation coefficient greater than the preset value as the target historical features.

4. The multivariate time series prediction method based on proximal strategy optimization according to claim 3, characterized in that, The step of performing feature extraction on the preprocessed historical data to construct historical features includes: A factorization machine and a deep neural network are pre-constructed, wherein the deep neural network and the factorization machine share the same embedding layer; The categorical features in the target historical features are obtained, the categorical features are converted into data codes, the data codes are input into the embedding layer, and the output of the embedding layer is obtained to complete the capture of categorical features. The embedding layer is used to map high-dimensional feature vectors into low-dimensional real number vectors and to convert sparse categorical features into dense embedding features. The target historical features are divided into different feature domains. The data encoding and the output of the embedding layer are input into a deep neural network to obtain a first output value. The feature data of different feature domains are input into a factorization machine to obtain a second output value. The cross features of the time series are obtained based on the first output value and the second output value. Each feature domain corresponds to a different sparse embedding layer and a dense embedding layer. Based on the captured categorical features and the cross features of the time series, the historical features are constructed.

5. The multivariate time series prediction method based on proximal strategy optimization according to claim 4, characterized in that, The definition of reinforcement learning elements for multivariate time series prediction, and the construction of a proximal policy optimization model based on these elements, the proximal policy optimization model including a policy network and a value function network, including: The state elements, action elements, reward elements, and strategy elements of the multivariate time series prediction are defined. Construct a policy network, which takes the environment state as input and outputs the probability distribution of all action elements under the current policy element; Construct a value function network, which is used to take the environmental state as input and output an estimated value of the current environmental state; The policy network and the value function network constitute the proximal policy optimization model.

6. A multivariate time series forecasting system based on proximal strategy optimization, characterized in that, The system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, performs the following steps: Historical data is acquired, preprocessed, and feature extraction is performed on the preprocessed historical data to complete the construction of historical features. In the feature extraction operation, categorical features are captured based on word embedding, and cross features of time series are automatically learned based on deep learning and factorization machine. A reinforcement learning element for multivariate time series prediction is defined, and a proximal policy optimization model is constructed based on the reinforcement learning element. The proximal policy optimization model includes a policy network and a value function network. The loss function of the policy network adopts a truncated advantage function, and the policy network uses multiple mini-batch samples for policy updates. The near-end strategy optimization model is trained based on the historical features. After training, a multivariate time series prediction model is obtained. The system acquires real-time data to be predicted, preprocesses the real-time data, extracts features from the preprocessed real-time data to construct real-time data features, inputs the real-time data features into the multivariate time series prediction model, obtains the output of the multivariate time series prediction model, and completes the prediction of future time series.

7. The multivariate time series prediction system based on proximal strategy optimization according to claim 6, characterized in that, When the computer program is executed by the processor, it performs the following steps: Based on specific business processes or domain knowledge, historical data is obtained, and the historical data is the true value of the time series. After performing data cleaning and normalization operations on the historical data, preprocessed historical data is obtained.

8. The multivariate time series prediction system based on proximal strategy optimization according to claim 7, characterized in that, When the computer program is executed by the processor, it also performs the following steps: Based on the preprocessed historical data, time-series index features, statistical index features, volatility index features, slope index features, and regression index features are calculated. Calculate the variance inflation factor among the above-mentioned indicator features, and remove features whose variance factor value is greater than a preset threshold. Calculate the Pearson correlation coefficient between the above-mentioned indicator features and the predicted values, and take the features with a Pearson correlation coefficient greater than the preset value as the target historical features.

9. The multivariate time series prediction system based on proximal strategy optimization according to claim 8, characterized in that, When the computer program is executed by the processor, it also performs the following steps: A factorization machine and a deep neural network are pre-constructed, wherein the deep neural network and the factorization machine share the same embedding layer; The categorical features in the target historical features are obtained, the categorical features are converted into data codes, the data codes are input into the embedding layer, and the output of the embedding layer is obtained to complete the capture of categorical features. The embedding layer is used to map high-dimensional feature vectors into low-dimensional real number vectors and to convert sparse categorical features into dense embedding features. The target historical features are divided into different feature domains. The data encoding and the output of the embedding layer are input into a deep neural network to obtain a first output value. The feature data of different feature domains are input into a factorization machine to obtain a second output value. The cross features of the time series are obtained based on the first output value and the second output value. Each feature domain corresponds to a different sparse embedding layer and a dense embedding layer. Based on the captured categorical features and the cross features of the time series, the historical features are constructed.

10. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the multivariate time series forecasting method based on proximal policy optimization as described in any one of claims 1-5.

Images