Prediction model training method and device

Abstract

The present invention provides a prediction model training method and device, which can comprehensively improve the model's ability to capture key factors and learn historical data. The method includes: performing variable selection on the original input data set to obtain a selected variable feature set; determining short-term correlation according to a target selection feature vector; generating a unified time feature according to the short-term correlation; determining a static enhanced time feature corresponding to the unified time feature; performing sparse probabilistic self-attention mechanism processing on the static enhanced time feature to obtain a target time feature; performing nonlinear processing on the target time feature to obtain a nonlinear processing time feature; generating a gated residual connection result according to the nonlinear processing time feature and the unified time feature; generating a prediction result according to the gated residual connection result; iteratively executing the above steps based on the prediction result and the objective function until a preset iteration termination condition is reached to obtain a prediction model.

Description

[Technical field]

[0001] The invention belongs to the technical field of multivariate time series prediction, and in particular relates to a training method and device for a base prediction model. [Background technology]

[0002] Multivariate time series data is a set of random variables ordered by time. It is usually the result of observing a potential process at a given sampling rate in equally spaced time periods. Time series data essentially reflects the trend of one or more random variables changing over time, and the core of the time series prediction problem is to mine this law from the data and use it to estimate future data. Combining the attention mechanism with the recurrent neural network and applying it to multivariate time series prediction can improve the ability to capture key influencing factors, thereby improving the accuracy and interpretability of the prediction results.

[0003] Traditional time series forecasting methods are usually based on two assumptions: (1) the time series process is stationary, usually weakly stationary or second-order stationary. Even if it is not the case, it can be transformed to a stationary state; (2) the time series is the output of a linear filter, and the input of the filter is a purely random process, usually following a Gaussian distribution. Although these assumptions are reasonable, many time series in reality are neither stationary nor can they be described as linear processes. In addition, the vast majority of time series cannot be easily transformed into a stationary process.

[0004] Traditional time series prediction methods need to ensure the integrity of the data series, that is, there must be no missing values. However, the problems and systems that need to be solved today often contain a large amount of information, have very complex data relationships and great uncertainty. The model itself is easily restricted by ideal assumptions, lacks flexibility, and has low prediction accuracy. [Summary of the invention]

[0005] The present invention provides a prediction model training method, device and storage medium, which can comprehensively improve the model's ability to capture key factors and learn historical data.

[0006] A first aspect of the present invention provides a prediction model training method, comprising:

[0007] Perform variable selection on the original input data set to obtain a selected variable feature set;

[0008] Determining the short-term correlation between the mutation key information corresponding to the target selection feature vector and the target context content according to the target selection feature vector, wherein the target selection feature vector is any one of the selection feature vectors in the selection variable feature set, and the target context content is the context content corresponding to the target selection feature vector;

[0009] Generating a unified time feature corresponding to the target selection feature vector according to the short-term correlation;

[0010] Determining a static enhanced time feature corresponding to the unified time feature;

[0011] Processing the static reinforcement time feature with a sparse probabilistic self-attention mechanism to obtain a target time feature;

[0012] Performing nonlinear processing on the target time feature to obtain a nonlinear processed time feature;

[0013] Generate a gated residual connection result according to the nonlinear processing time feature and the unified time feature;

[0014] Generate a prediction result according to the gated residual connection result;

[0015] The above steps are iteratively performed based on the prediction results and the objective function until a preset iteration termination condition is reached to obtain a prediction model.

[0016] A second aspect of the present invention provides a prediction model training device, comprising:

[0017] A variable selection unit is used to select variables from the original input data set to obtain a selected variable feature set;

[0018] A short-term correlation determination unit, used to determine the short-term correlation between the mutation key information corresponding to the target selection feature vector and the target context content according to the target selection feature vector, wherein the target selection feature vector is any one of the selection feature vectors in the selection variable feature set, and the target context content is the context content corresponding to the target selection feature vector;

[0019] A time feature determination unit, configured to generate a unified time feature corresponding to the target selection feature vector according to the short-term correlation;

[0020] A time feature enhancement unit, used to determine the static enhanced time feature corresponding to the unified time feature;

[0021] A time feature processing unit, used for performing sparse probabilistic self-attention mechanism processing on the static reinforcement time feature to obtain a target time feature;

[0022] A nonlinear processing unit, used for performing nonlinear processing on the target time feature to obtain a nonlinear processed time feature;

[0023] A gated residual connection unit, used to generate a gated residual connection result according to the nonlinear processing time feature and the unified time feature;

[0024] A prediction unit, configured to generate a prediction result according to the gated residual connection result;

[0025] The training unit is used to iteratively execute the above steps based on the prediction results and the objective function until a preset iteration termination condition is reached to obtain a prediction model.

[0026] The third aspect of the present invention provides a computer device comprising at least one connected processor, memory and transceiver, wherein the memory is used to store program code, and the processor is used to call the program code in the memory to execute the steps of the prediction model training method described in the first aspect above.

[0027] A fourth aspect of the present invention provides a computer storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the steps of the prediction model training method described in any of the above aspects.

[0028] Compared with the related art, in the embodiment provided by the present invention, through the variable selection network and the sparse probabilistic self-attention mechanism, the key factors affecting the prediction are captured through the two-stage key factor identification and selection. And by learning the short-term correlation of the time series and combining the attention mechanism to obtain the long-term correlation of the time series, efficient learning of historical time series data information is achieved. By comprehensively improving the model's ability to capture key factors and learn historical data, the technical effect of improving the accuracy and interpretability of the prediction results can be achieved.

Brief Description of the Drawings

[0029] Figure 1 A schematic diagram of a flow chart of a method for training a prediction model provided by an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of the architecture of a prediction model provided by an embodiment of the present invention;

[0031] Figure 3 This is a comparison chart of the importance of influencing factors generated by the sparse probabilistic self-attention mechanism in an embodiment of the present invention for real data;

[0032] Figure 4 A schematic diagram of prediction results of the prediction model of the present invention and all baseline models in a Bike dataset experiment provided by an embodiment of the present invention;

[0033] Figure 5 A virtual structural diagram of a training device for a prediction model provided by an embodiment of the present invention;

[0034] Figure 6 A schematic diagram of the hardware structure of a server provided in an embodiment of the present invention. [Specific implementation method]

[0035] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0036] With the development of artificial intelligence technology, intelligent algorithms such as neural networks (ANN), support vector machines (SVM), and Gaussian processes (GP) are often used for time series prediction, providing new methods and ideas for solving nonlinear prediction problems. However, these prediction methods and models have the disadvantages of being relatively simple, having their own characteristics and applicable scenarios, being unable to fully explore the original data information, and having low prediction accuracy.

[0037] In recent years, due to the success of attention-based methods in other related fields, attention mechanisms have gradually been applied to the field of time series prediction. However, many previous attention-based methods did not take into account the different types of input data in multivariate time series and the differences in importance. They simply connected the relevant features together at each step, and could not effectively screen the key factors affecting the prediction, resulting in low prediction accuracy. At the same time, mutation information is also key information that is often ignored in multivariate time series prediction, which further limits the improvement of prediction accuracy.

[0038] The following describes the prediction model training method from the perspective of a prediction model training device. The prediction model training device may be a server or a service unit in the server, and is not specifically limited.

[0039] Please refer to Figure 1 , Figure 1 A flowchart of a method for training a prediction model provided by an embodiment of the present invention includes:

[0040] 101. Perform feature transformation on the original input data set to obtain a selected variable feature set.

[0041] In this embodiment, the training device of the prediction model can perform feature transformation on the original input data set to obtain a selection variable feature set. For ease of understanding, the following is combined with Figure 2 For a description of feature transformation, see Figure 2 , Figure 2 A structural diagram of a prediction model provided for an embodiment of the present invention, the prediction model includes an encoder combining a variable selection network based on a gating mechanism, a static covariate encoder and a GRU encoder-decoder, a decoder focusing on a sparse probabilistic self-attention mechanism, and a new objective function combining root mean square error, mean absolute error and second-order difference.

[0042] The original input data set may include three different types of data depending on the prediction type: static variables, historical data, and future known data. Not all data sets contain static variables. The bicycle data set is used as an example. The bicycle data set includes the number of public bicycles rented per hour from December 2017 to November 2018 in a certain bicycle sharing system, as well as the corresponding weather data and holiday information. The number of public bicycles rented is the target sequence (that is, the time series to be predicted), and other time series include weather data (the weather data includes but is not limited to temperature, humidity, wind speed, visibility, dew point, solar radiation intensity, snowfall, rainfall, etc.), seasonal information, and holiday information, among which:

[0043] S (static): A static variable refers to a variable that does not change over time. Not all datasets contain static variables. When a dataset does not include static variables, all context vectors related to the static variables used in the model training process are assigned a value of 0.

[0044] P(past): Historical data refers to the historical data of all non-static variables, including the historical data corresponding to the number of public bicycle rentals, weather data, seasonal information, and holiday information.

[0045] F(future): future known data, i.e., other sequences except the target sequence. In the experiment, in order to verify the accuracy of the prediction model, the values ​​of non-predicted variables in the predicted time period are known to improve the prediction accuracy. In practical applications, future known data can be obtained. For example, weather data in future known data can be obtained in advance through meteorological forecasts, and seasonal information and holiday information can also be obtained in advance. The data of these non-predicted target sequences in the prediction time period are regarded as future known data.

[0046] First, for P data, that is, historical data, variable selection (that is, feature transformation) is performed to obtain the corresponding selected feature vector. Specifically, the following formula can be used to perform variable selection on the original input data set to obtain the selected variable feature set:

[0047]

[0048] in, is the selected feature vector at time t in the selected variable feature set, is the transformed feature vector of the jth variable in the original input data set at time t. Each variable in the original input data set corresponds to a corresponding weight is the weight vector of the original data set at time t, μ xt Gated Residual Network, Δt is the selected variable feature set of the transformed original data set at time t, c s represents the context vector, yes The j-th element of .

[0049] That is, firstly, a variable selection network is used to select individual variables for the original input data, where the original data includes categorical variables and continuous variables, entity embedding is used for categorical variables, and linear variables are used for input continuous variables. That is, each input variable is converted into d model dimensional feature vector, input the corresponding gated residual network for nonlinear processing. Specifically, each input variable can be processed nonlinearly using the following formula: At the same time, the transformed eigenvector Δ t With the context vector c s Feed it into the Gated Residual Network (GRN), and then use the Softmax function to calculate the variable selection weights The nonlinearly processed features are weighted based on the variable selection weights using the following formula to perform variable selection:

[0050]

[0051] It should be noted that the context vector can be generated by a static covariate encoder. Specifically, the static variables after variable selection can be input into the static covariate encoder, and four independent gated residual networks (GRNs) are used to generate different context vectors c s 、c c 、c h 、c e It can be understood that for variable selection of static variables in the original input dataset, the context vector c s Omitted because static information is already accessible.

[0052] It should also be noted that since too large or too small time granularity is not conducive to the training of deep learning models and time series prediction, if the original input data set is a complete data set, no data preprocessing is required. If the original data is an incomplete data set, the original input data set needs to be preprocessed and the appropriate time granularity needs to be selected. The preprocessing includes but is not limited to: interpolation processing of the original input data set, for example, by interpolating the average value of the data within 1 hour, the time step of the data set is converted to 1 hour. When the data set is interpolated, some virtual data will be generated based on smoothing, which will reduce the authenticity of the information data, so corresponding smoothing processing is required. The main purpose of data preprocessing is to make the data set data continuous in time and the data time step is the same, that is, to fill in the null values ​​and unify the time step.

[0053] It should be noted that the above-mentioned method for selecting variables of historical data P is described in detail. The method for selecting variables of static variables S and future known data F is similar to that for selecting variables of historical data P, and will not be described in detail here.

[0054] 102. Determine the short-term correlation between the mutation key information corresponding to the target selection feature vector and the target context content according to the target selection feature vector.

[0055] In this embodiment, the training device of the prediction model can determine the short-term correlation between the mutation key information corresponding to the target selection feature vector and the target context content based on the target selection feature vector, wherein the target selection feature vector is any selection feature vector in the selection variable feature set, and any one here refers to a feature vector selected when traversing the selection feature vectors in the selection variable feature set, and the target context content is the context content corresponding to the target selection feature vector. Specifically, the target selection feature vector can be input into the GRU encoder, and the target selection feature vector includes the selection feature vector corresponding to the historical data and the selection feature vector corresponding to the known future data to extract the short-term correlation between the mutation key information and the target context content.

[0056] It should be noted that the mutation key information refers to the mutation data in the data, and the context content refers to the data itself and other data within a certain time interval before and after the mutation data. For example, the mutation of the number of bicycle rentals may be due to time, that is, the change itself shows a certain regularity (more rentals during peak hours in the morning and evening), or due to weather reasons (such as rain leading to a decrease in rentals), or due to holidays (holidays lead to an increase in the number of travelers and an increase in the number of bicycle rentals).

[0057] 103. Generate a unified time feature corresponding to the target selection feature vector based on the short-term correlation.

[0058] In this embodiment, the training device of the prediction model can generate the unified time feature corresponding to the target selection feature vector according to the short-term correlation. Specifically, the unified time feature corresponding to the target selection feature vector can be determined by the following formula:

[0059]

[0060] in, To unify the time characteristics, is the short-term correlation, LayerNorm is the layer normalization operation, for Gated Linear Unit, The feature vector generated by the variable selection network for historical data and future known data, m is the position index, m∈[-k, τ max ]That is, the prediction reference interval and the prediction interval.

[0061] 104. Determine the static reinforcement time characteristics corresponding to the unified time characteristics.

[0062] In this embodiment, the training device of the prediction model inputs the unified time feature into the GRN, and fuses it with the context vector generated by the static covariate encoder to obtain the static enhanced time feature. Specifically, the static time enhanced feature can be determined by the following formula:

[0063]

[0064] Among them, v(t,m) is the static time enhancement feature, GRN v is the v-gated residual network, is the unified time feature, c e The context vector is the static variable after variable selection, which is input into the static covariate encoder. Four independent gated residual networks (GRNs) are used to generate different context vectors c s 、c c 、c h 、c e That is, the training device of the prediction model can first obtain four different context vectors, and when the corresponding context vector needs to be used during the training process, it can be directly used.

[0065] 105. Static reinforcement time features are processed by sparse probabilistic self-attention mechanism to obtain target time features.

[0066] In this embodiment, in the decoder focusing on the sparse probabilistic self-attention mechanism, a series of components are used to further improve the performance of the model in learning long-term correlations and capturing key factors. The training device of the prediction model can input static reinforced time features into the sparse probabilistic self-attention mechanism to obtain target time features. While learning the long-term correlation of the time series, the prediction model's attention to the key factors affecting the prediction is further enhanced. It can be understood that the training device of the prediction model inputs the static reinforced time features into the sparse probabilistic self-attention mechanism, and outputs the time features processed by the sparse probabilistic self-attention mechanism. The time features exist in the form of key-value pairs. In this embodiment, only key key-value pairs are retained, and the remaining non-key key-value pairs are assigned 0. Specifically:

[0067] The target time characteristics are determined by the following formula:

[0068] Φ(t)=ProbSparseAttention(Υ(t),Υ(t),Υ(t));

[0069] Among them, Φ(t) is the target time feature, ProbSparseAttention is the coefficient probability self-attention mechanism, and Υ(t) is the matrix composed of static reinforcement time features.

[0070] It should be noted that in the sparse probabilistic self-attention mechanism, the input Query, Key, and Value vectors are calculated using the probabilistic sparsity metric to filter out the key key-value pairs that contribute the main attention. The larger the calculated value obtained using the probabilistic sparsity metric, the greater the probability of containing important dot product pairs of the self-attention mechanism, that is, the key factors affecting the prediction. Then, the attention mechanism is implemented by allowing each key to focus only on the most important part of the query. The following formula is used to enhance the screening of key factors, suppress the influence of unimportant factors, and reduce the time complexity and memory usage of the model:

[0071]

[0072]

[0073] Among them, q i , k i 、v i Represents the i-th row in the Query, Key, and Value vectors, respectively. K represents the input length of the Key vector, and d represents the input dimension; is a sparse matrix of the same size as q, containing only the top-n queries under the probabilistic sparsity measure. Figure 3 , Figure 3A comparison chart of the importance of influencing factors generated by the sparse probabilistic self-attention mechanism in an embodiment of the present invention for real data is given in FIG.

[0074] 106. Perform nonlinear processing on the target time feature to obtain a nonlinear processed time feature.

[0075] In this embodiment, the training device of the prediction model can determine the nonlinear processing time characteristics by the following formula:

[0076] ω(t,m)=GRN ω (ρ(t,m));

[0077] Among them, ω(t,m) is the nonlinear processing time feature, GRN ω For the ω-gated residual network, ρ(t,m) is determined by the following formula:

[0078] ρ(t,m)=LayerNorm(v(t,m)+GLU ρ (φ(t,m)));

[0079] Among them, LayerNorm is the gated layer, v(t,m) is the static time enhancement feature, GLU ρ is a ρ gated linear unit, and φ(t,m) is the vector in the target temporal feature Φ(t).

[0080] 107. Generate gated residual connection results based on nonlinear processing time features and unified time features.

[0081] In this embodiment, the training device of the prediction model uses a gated residual connection that skips the entire decoder module after the decoder, and generates a gated residual connection result according to the nonlinear processing time feature and the unified time feature through the following formula:

[0082]

[0083] in, is the gated residual connection result, LayerNorm is the gated layer, To unify the time characteristics, for Gated linear unit, ω(t,m) is the nonlinear processing time feature.

[0084] 108. Generate prediction results based on the gated residual connection results,

[0085] In this embodiment, the prediction result is generated by the following formula:

[0086]

[0087] in, For the prediction result, W q is the weight of the specified quantile q, b q is the linear coefficient for the specified quantile q, It is the result of gated residual connection.

[0088] 109. Iteratively execute the above steps based on the prediction result and the objective function until a preset iteration termination condition is reached to obtain a prediction model.

[0089] In this embodiment, after obtaining the prediction result, the training device of the prediction model can iteratively perform the above steps based on the prediction result and the objective function (loss function) until the preset iteration termination condition is reached to obtain the prediction model. The root mean square error and the mean absolute error are combined in the objective function to comprehensively consider the error amplitude and mutation key information between the predicted value and the true value. At the same time, the second-order difference operation is integrated to further guide the model to learn the mutation key information in the time series data. The objective function formula is as follows:

[0090]

[0091]

[0092] Among them, x k,t is the predicted value of the kth sequence at time t, u k,t is the true value of the kth sequence at time t, C is the fusion proportional constant of the root mean square error and the mean absolute error, is the second-order difference operation, |.| is the absolute value operation, λ is the hyperparameter λ, T is the prediction time period, and K is the time series data set corresponding to T.

[0093] The verification of the prediction model is explained below. The present invention verifies the effectiveness of the proposed model by selecting 4 real multivariate time series data sets from different fields. Each selected data set has complex time correlation and rich time change information, and can truly reflect the common data conditions in the multivariate time series prediction problem.

[0094] Seoul Bike Sharing Demand This dataset contains the number of public bicycles rented per hour from December 2017 to November 2018 in a certain bicycle sharing system, as well as the corresponding weather data and holiday information. The number of public bicycles rented is the target sequence, and the exogenous time series includes weather data (temperature, humidity, wind speed, visibility, dew point, solar radiation intensity, snowfall, rainfall), seasonal information, and holiday information. Use the data of the past 7 days (i.e. 168 hours) to predict the number of public bicycles rented in the next 1 day (i.e. 24 hours).

[0095] Metro Traffic Volume: This dataset contains hourly traffic volume and corresponding weather characteristics and holiday information of a certain place from 2012 to 2018. The target sequence for prediction is traffic volume, and weather characteristics include temperature, cloud cover, etc. Use 7 days of historical data to predict the traffic volume for the next day.

[0096] PM2.5 (Beijing PM2.5 Index) This dataset contains hourly PM2.5 index data of a certain place from 2010 to 2014 and a series of meteorological data related to air quality. The air PM2.5 index is the target sequence, and the main meteorological attributes include wind speed, temperature, air pressure, rain and snow conditions, etc. This paper uses 3 days of historical data to predict the air quality index for the next day.

[0097] Gold (Gold Price) This dataset contains daily gold ETF data and other international financial market data from 2012 to 2018. The prediction target sequence is the adjusted closing price of the gold ETF. Other international financial market information as exogenous time series mainly includes data such as crude oil prices, the Standard & Poor's (S&P) 500 Index, the Dow Jones Index, the euro-dollar exchange rate, and multiple metal prices. The present invention uses the data of the past 10 trading days to predict the closing price of the gold ETF in the next week (5 trading days).

[0098] The evaluation indicators used are the mean absolute error (MAE) and root mean square error (RMSE), which are widely used in time series forecasting research and are defined as follows:

[0099]

[0100] Among them, y i and Represents the true value and model predicted value in the multivariate time series forecasting task, and n represents the total number of samples in the test set. For MAE and RMSE, the lower the value, the higher the forecast accuracy.

[0101] Table 1

[0102]

[0103] From Table 1, it can be observed that the proposed method significantly improves the prediction performance in the prediction tasks of the three data sets and achieves the best prediction effect; on the PM2.5 index data set, the DA-RNN model using the single-step prediction method achieves the best prediction effect, but among all the multi-step prediction methods, the prediction model trained by the prediction model training method provided by the present invention still reaches the optimal prediction accuracy. Further in-depth analysis of the experimental results shows that, first, on different data, the proposed method shows better prediction accuracy than the recurrent neural network. In the Bike data set, the root mean square error of the method of the present invention is reduced by 50.3% compared with the optimal RNN model (DeepAR); in the Metro data set, the root mean square error of the method of the present invention is reduced by 57.6% compared with the optimal RNN model (DA-RNN); in the Gold data set, the root mean square error of the method of the present invention is reduced by 8.3% compared with the optimal RNN model (N-BEATS). This shows that compared with the RNN-based model, the self-attention mechanism combined in the method proposed by the present invention produces better prediction performance. Secondly, in the prediction results of all data sets, the prediction accuracy of the proposed method is better than that of the TFT model. In the long-term prediction task of the Bike data set, the root mean square error of the method of the present invention is reduced by 22.1%, and in the short-term prediction task of the Gold data set, the root mean square error is reduced by 45%, proving that the dual-stage key factor identification selection and the learning of mutation key information improve the prediction accuracy of the model. Finally, the proposed method achieves the minimum level of performance fluctuation in all prediction tasks of different data set sizes, indicating that the method has strong stability and robustness.

[0104] Figure 4 The prediction results of the proposed method and all baseline models in the Bike dataset experiment are shown. Figure 4 The solid line with squares in the figure corresponds to the real value of the demand for shared bicycles in the next 24 hours, and the dotted line with dots corresponds to the predicted value generated by the prediction model obtained by the training method of the prediction model provided by the present invention, and the remaining dotted lines in the figure are the prediction results of the experimental control model. As can be seen from the figure, the prediction of the prediction model generated by the training method of the prediction model provided by the present invention can best fit the real value curve. Except that the predicted values ​​at 2-3 time points have a certain visible deviation from the real values, the remaining predicted values ​​are basically consistent with the actual values. In particular, the model can accurately predict the demand at the time stage of 5-10h and 16-20h when obvious mutations occur. This shows that the new objective function of learning periodic laws and using attention to time change information by fusing time information through the GRU codec and the self-attention mechanism brings better periodic data and mutation data prediction capabilities to the model, thereby improving the prediction accuracy.

[0105] In summary, it can be seen that in the embodiments provided by the present invention, by fusing the attention mechanism with the recurrent neural network, applying the variable selection network based on the gating mechanism and the sparse probabilistic self-attention mechanism, the key factors affecting the prediction are captured through the two-stage key factor identification and selection. The GRU encoder-decoder is used to learn the short-term correlation of the time series, and the attention mechanism is combined to obtain the long-term correlation of the time series, so as to realize efficient learning of historical time series data information. A new objective function based on mean absolute error, root mean square error and second-order difference operation is used to effectively guide the model to learn the key information of mutations in time series data. By comprehensively improving the model's ability to capture key factors and learn historical data, the technical effect of improving the accuracy and interpretability of the prediction results can be achieved.

[0106] The present invention is described above from the perspective of a training method for a prediction model. The present invention is described below from the perspective of a training device for a prediction model.

[0107] See also Figure 5 , Figure 5 A virtual structural diagram of a prediction model training device provided by an embodiment of the present invention, the prediction model training device 500 includes:

[0108] The variable selection unit 501 is used to perform variable selection on the original input data set to obtain a selected variable feature set;

[0109] A short-term correlation determination unit 502 is used to determine the short-term correlation between the mutation key information corresponding to the target selection feature vector and the target context content according to the target selection feature vector, wherein the target selection feature vector is any one of the selection feature vectors in the selection variable feature set, and the target context content is the context content corresponding to the target selection feature vector;

[0110] A time feature determination unit 503, configured to generate a unified time feature corresponding to the target selection feature vector according to the short-term correlation;

[0111] A time feature enhancement unit 504, configured to determine a static enhanced time feature corresponding to the unified time feature;

[0112] A time feature processing unit 505 is used to perform sparse probabilistic self-attention mechanism processing on the static enhanced time feature to obtain a target time feature;

[0113] A non-linear processing unit 506, configured to perform non-linear processing on the target time feature to obtain a non-linear processed time feature;

[0114] A gated residual connection unit 507, configured to generate a gated residual connection result according to the nonlinear processing time feature and the unified time feature;

[0115] A prediction unit 508, configured to generate a prediction result according to the gated residual connection result;

[0116] The training unit 509 is used to iteratively execute the above steps based on the prediction results and the objective function until a preset iteration termination condition is reached to obtain a prediction model.

[0117] In a possible design, the variable selection unit 501 is specifically used for:

[0118] The selected variable feature set is determined by the following formula:

[0119]

[0120] in, is the selected feature vector at time t in the selected variable feature set, is the transformed feature vector of the jth variable in the original input data set at time t, and each variable in the original input data set corresponds to a corresponding weight is the weight vector of the original data set at time t, for Gated Residual Network, Δ t is the selected variable feature set of the transformed original data set at time t, c s represents the context vector, yes The j-th element of .

[0121] In a possible design, the time feature determination unit 503 is specifically used to:

[0122] The unified time feature is determined by the following formula:

[0123]

[0124] in, is the unified time feature, is the short-term correlation, LayerNorm is the layer normalization operation, for Gated Linear Unit, is the selected feature vector at the mth position in the selected variable feature set, m∈[-k, τ max ].

[0125] In a possible design, the time feature enhancement unit 504 is specifically used for:

[0126] The static time-intensive characteristic is determined by the following formula:

[0127]

[0128] Where v(t,m) is the static time-enhanced feature, GRN v is the v-gated residual network, is the unified time feature, c e is the context vector.

[0129] In a possible design, the time feature processing unit 505 is specifically used for:

[0130] The target time characteristic is determined by the following formula:

[0131] Φ(t)=ProbSparseAttention(Υ(t),Υ(t),Υ(t));

[0132] Among them, Φ(t) is the target time feature, ProbSparseAttention is the coefficient probability self-attention mechanism, and Υ(t) is the matrix composed of the static reinforcement time feature.

[0133] In a possible design, the nonlinear processing unit 506 is specifically used for:

[0134] The nonlinear processing time characteristic is generated by the following formula:

[0135] ω(t,m)=GRN ω (ρ(t,m));

[0136] Among them, ω(t,m) is the time feature of the nonlinear processing, GRN ω For the ω-gated residual network, ρ(t,m) is determined by the following formula:

[0137] ρ(t,m)=LayerNorm(v(t,m)+GLU ρ (φ(t,m)));

[0138] Among them, LayerNorm is the gated layer, v(t,m) is the static time enhancement feature, GLU ρ is a ρ gated linear unit, and φ(t,m) is a vector in the target time feature Φ(t).

[0139] In a possible design, the gated residual connection unit 507 is specifically used for:

[0140] The gated residual connection result is determined by the following formula:

[0141]

[0142] in, is the gated residual connection result, LayerNorm is the gated layer, is the unified time feature, for Gated linear unit, ω(t,m) is the nonlinear processing time feature.

[0143] In a possible design, the prediction unit 508 is specifically used for:

[0144] The prediction result is generated by the following formula:

[0145]

[0146] in, is the prediction result, W q is the weight of the specified quantile q, b q is the linear coefficient for the specified quantile q, is the gated residual connection result.

[0147] In one possible design, the objective function is defined as follows:

[0148]

[0149]

[0150] Among them, x k,t is the predicted value of the kth sequence at time t, u k,t is the true value of the kth sequence at time t, C is the fusion proportional constant of the root mean square error and the mean absolute error, is a second-order difference operation, |.| is an absolute value operation, λ is a hyperparameter λ, T is the prediction time period, and K is the time series data set corresponding to T.

[0151] Figure 6 It is a structural diagram of the server of the present invention, such as Figure 6As shown, the server 600 of this embodiment includes at least one processor 601, at least one network interface 604 or other user interface 603, a memory 605, and at least one communication bus 602. The server 600 optionally includes a user interface 603, including a display, a keyboard or a clicking device. The memory 605 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 605 stores execution instructions. When the server 600 is running, the processor 601 communicates with the memory 605, and the processor 601 calls the instructions stored in the memory 605 to execute the above-mentioned prediction model training method. The operating system 606 includes various programs for implementing various basic businesses and processing tasks based on hardware.

[0152] The server provided in the embodiment of the present invention can execute the technical solution of the embodiment of the above-mentioned prediction model training method. Its implementation principle and technical effect are similar and will not be repeated here.

[0153] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the method flow related to the training device of the prediction model in any of the above method embodiments is implemented. Correspondingly, the computer can be the training device of the above prediction model.

[0154] The embodiment of the present invention further provides a computer program or a computer program product including the computer program, which, when executed on a computer, enables the computer to implement the method flow related to the training device of the prediction model in any of the above method embodiments. Correspondingly, the computer can be the above prediction model training device.

[0155] In the above Figure 1 In the corresponding embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0156] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a prediction model, wherein the prediction model is used in a server in artificial intelligence technology, or a service unit in the server, and the prediction model is used to predict the number of shared bicycle rentals, subway traffic volume, or PM2.5 index, characterized in that: include: Variable selection is performed on the original input data set to obtain a selected variable feature set, wherein the original input data set includes at least one of static variables, historical data or future known data; the static variable is a variable that does not change over time; the historical data refers to the historical data of all non-static variables, including historical data corresponding to at least one of the number of public bicycle rentals, weather data, seasonal information and holiday information; the future known data includes weather data obtained in advance through meteorological forecasts, and / or seasonal information, and / or holiday information; Determine the short-term correlation between the mutation key information corresponding to the target selection feature vector and the target context content according to the target selection feature vector, wherein the target selection feature vector is any one of the selection feature vectors in the selection variable feature set, and the target context content is the context content corresponding to the target selection feature vector; the mutation key information includes the mutation data in the data, and the context content of the mutation data is the data itself and other data within a certain time interval before and after the mutation data; when the prediction model is used to predict the number of shared bicycle rentals, the mutation data includes the mutation of the number of bicycle rentals, and the mutation of the number of bicycle rentals is caused by time reasons, weather reasons or holiday reasons; the time reasons include an increase in the number of bicycle rentals during peak hours in the morning and evening, the weather reasons include a decrease in the number of bicycle rentals due to rain, and the holiday reasons include an increase in the number of travelers due to holidays, which increases the number of bicycle rentals; Generating a unified time feature corresponding to the target selection feature vector according to the short-term correlation; Determining a static enhanced time feature corresponding to the unified time feature; Processing the static reinforcement time feature with a sparse probabilistic self-attention mechanism to obtain a target time feature; Performing nonlinear processing on the target time feature to obtain a nonlinear processed time feature; Generate a gated residual connection result according to the nonlinear processing time feature and the unified time feature; Generate a prediction result according to the gated residual connection result; The above steps are iteratively performed based on the prediction results and the objective function until a preset iteration termination condition is reached to obtain a prediction model; wherein, When the prediction model is used to predict the number of shared bicycles rented, the original input data set includes the number of bicycles rented by the bicycle sharing system in the past preset time period and the corresponding weather data and holiday information, the number of bicycles rented is the target sequence, and the exogenous time series includes weather data, seasonal information and holiday information; the data of the past preset time period is used to predict the number of shared bicycles rented in the future preset time period; When the prediction model is used to predict subway traffic volume, the original input data set includes the hourly traffic volume of the target location in the past preset period and the corresponding weather characteristic information and holiday information; the prediction target sequence is the traffic volume, and the weather characteristic information includes temperature and / or cloud cover; the historical data of the past preset period is used to predict the traffic volume in the future preset period; When the prediction model is used to predict PM2.5, the original input data set includes hourly PM2.5 index data of the target location within a preset period in the past and meteorological data related to air quality; the air PM2.5 index is a target sequence, and the meteorological attributes include at least one of wind speed, temperature, air pressure or rain and snow conditions; the historical data of the preset period in the past is used to predict the air quality index of the preset period in the future. The step of performing variable selection on the original input data set to obtain a selected variable feature set includes: The selected variable feature set is determined by the following formula: in, is the selected feature vector at time t in the selected variable feature set, is the transformed feature vector of the jth variable in the original input data set at time t, and each variable in the original input data set corresponds to a corresponding weight is the weight vector of the original input data set at time t, for Gated Residual Network, Δ t is the selected variable feature set of the transformed original input data set at time t, c s represents the context vector, yes The jth element of The generating of the unified time feature corresponding to the target selection feature vector according to the short-term correlation comprises: The unified time feature is determined by the following formula: in, is the unified time feature, is the short-term correlation, LayerNorm is the layer normalization operation, for Gated Linear Unit, is the selected feature vector at the mth position in the selected variable feature set, m∈[-a, τ max ], The objective function is defined as follows: Among them, x k,t is the predicted value of the kth sequence at time t, u k,t is the true value of the kth sequence at time t, C is the fusion proportional constant of the root mean square error and the mean absolute error, is a second-order difference operation, |.| is an absolute value operation, λ is a hyperparameter λ, T is the prediction time period, and K is the time series data set corresponding to T.

2. The method according to claim 1, characterized in that Determining the static reinforcement time feature corresponding to the unified time feature includes: The static strengthening time characteristic is determined by the following formula: Where v(t,m) is the static reinforcement time characteristic, GRN v is the v-gated residual network, is the unified time feature, c e is the context vector.

3. The method according to claim 1, characterized in that The static enhanced time feature is processed by a sparse probabilistic self-attention mechanism to obtain a target time feature, including: The target time characteristic is determined by the following formula: Φ(t)=ProbSparseAttention(Υ(t),Υ(t),Υ(t)); Among them, Φ(t) is the target time feature, ProbSparseAttention is the sparse probabilistic self-attention mechanism, and Υ(t) is the matrix composed of the static reinforcement time feature.

4. The method according to claim 1, characterized in that: The performing nonlinear processing on the target time feature to obtain the nonlinear processed time feature comprises: The nonlinear processing time characteristic is generated by the following formula: ω(t,m)=GRN ω (ρ(t,m)); Among them, ω(t,m) is the time feature of the nonlinear processing, GRN ω For the ω-gated residual network, ρ(t,m) is determined by the following formula: ρ(t,m)=LayerNorm(v(t,m)+GLU ρ (φ(t,m))); Among them, LayerNorm is the gated layer, v(t,m) is the static enhanced time feature, GLU ρ is a ρ gated linear unit, and φ(t,m) is a vector in the target time feature Φ(t).

5. The method according to claim 1, characterized in that Generating a gated residual connection result according to the nonlinear processing time feature and the unified time feature comprises: The gated residual connection result is determined by the following formula: in, is the gated residual connection result, LayerNorm is the gated layer, is the unified time feature, for Gated linear unit, ω(t,m) is the nonlinear processing time feature.

6. The method according to claim 1, characterized in that Generating a prediction result according to the gated residual connection result comprises: The prediction result is generated by the following formula: in, is the prediction result, W q is the weight of the specified quantile q, b q is the linear coefficient for the specified quantile q, is the gated residual connection result.

7. A training device for a prediction model, wherein the prediction model is used in a server in artificial intelligence technology, or a service unit in the server, and the prediction model is used to predict the number of shared bicycle rentals, subway traffic volume, or PM2.5 index, characterized in that: include: A variable selection unit is used to perform variable selection on an original input data set to obtain a selected variable feature set, wherein the original input data set includes at least one of static variables, historical data or future known data; the static variable is a variable that does not change over time; the historical data refers to the historical data of all non-static variables, including historical data corresponding to at least one of the number of public bicycle rentals, weather data, seasonal information and holiday information; the future known data includes weather data obtained in advance through meteorological forecasts, and / or seasonal information, and / or holiday information; A short-term correlation determination unit is used to determine the short-term correlation between the mutation key information corresponding to the target selection feature vector and the target context content according to the target selection feature vector, wherein the target selection feature vector is any one of the selection feature vectors in the selection variable feature set, and the target context content is the context content corresponding to the target selection feature vector; the mutation key information includes mutation data in the data, and the context content of the mutation data is the data itself and other data within a certain time interval before and after the mutation data; when the prediction model is used to predict the number of shared bicycle rentals, the mutation data includes a mutation in the number of bicycle rentals, and the mutation in the number of bicycle rentals is caused by time reasons, weather reasons or holiday reasons; the time reasons include an increase in the number of bicycle rentals during peak hours in the morning and evening, the weather reasons include a decrease in the number of bicycle rentals due to rain, and the holiday reasons include an increase in the number of travelers due to holidays, which increases the number of bicycle rentals; A time feature determination unit, configured to generate a unified time feature corresponding to the target selection feature vector according to the short-term correlation; A time feature enhancement unit, used to determine the static enhanced time feature corresponding to the unified time feature; A time feature processing unit, used for performing sparse probabilistic self-attention mechanism processing on the static reinforcement time feature to obtain a target time feature; A nonlinear processing unit, used for performing nonlinear processing on the target time feature to obtain a nonlinear processed time feature; A gated residual connection unit, used to generate a gated residual connection result according to the nonlinear processing time feature and the unified time feature; A prediction unit, configured to generate a prediction result according to the gated residual connection result; A training unit is used to iteratively execute the above steps based on the prediction results and the objective function until a preset iteration termination condition is reached to obtain a prediction model; wherein, When the prediction model is used to predict the number of shared bicycles rented, the original input data set includes the number of bicycles rented by the bicycle sharing system in the past preset time period and the corresponding weather data and holiday information, the number of bicycles rented is the target sequence, and the exogenous time series includes weather data, seasonal information and holiday information; the data of the past preset time period is used to predict the number of shared bicycles rented in the future preset time period; When the prediction model is used to predict subway traffic volume, the original input data set includes the hourly traffic volume of the target location in the past preset period and the corresponding weather characteristic information and holiday information; the prediction target sequence is the traffic volume, and the weather characteristic information includes temperature and / or cloud cover; the historical data of the past preset period is used to predict the traffic volume in the future preset period; When the prediction model is used to predict PM2.5, the original input data set includes hourly PM2.5 index data of the target location in the past preset period and meteorological data related to air quality; the air PM2.5 index is a target sequence, and the meteorological attributes include at least one of wind speed, temperature, air pressure or rain and snow conditions; the historical data of the past preset period is used to predict the air quality index of the future preset period; The step of performing variable selection on the original input data set to obtain a selected variable feature set includes: The selected variable feature set is determined by the following formula: in, is the selected feature vector at time t in the selected variable feature set, is the transformed feature vector of the jth variable in the original input data set at time t, and each variable in the original input data set corresponds to a corresponding weight is the weight vector of the original input data set at time t, for Gated Residual Network, Δ t is the selected variable feature set of the transformed original input data set at time t, c s represents the context vector, is μ xt The jth element of The generating of the unified time feature corresponding to the target selection feature vector according to the short-term correlation comprises: The unified time feature is determined by the following formula: in, is the unified time feature, is the short-term correlation, LayerNorm is the layer normalization operation, for Gated Linear Unit, is the selected feature vector at the mth position in the selected variable feature set, m∈[-a, τ max ], The objective function is defined as follows: Among them, x k,t is the predicted value of the kth sequence at time t, u k,t is the true value of the kth sequence at time t, C is the fusion proportional constant of the root mean square error and the mean absolute error, is a second-order difference operation, |.| is an absolute value operation, λ is a hyperparameter λ, T is the prediction time period, and K is the time series data set corresponding to T.

Images