Time sequence prediction method based on time sequence prediction model and time sequence prediction model

Abstract

According to the time sequence prediction method based on the time sequence prediction model, the time sequence prediction model is a time sequence prediction model based on a deep learning network and comprises a coding network and a prediction network; the method comprises the following steps: acquiring a time sequence corresponding to a historical time period; and a target covariable sequence corresponding to the target time period; inputting the time sequence into a coding network, and coding data contained in the time sequence into a feature representation form by the coding network to obtain a feature representation sequence corresponding to the time sequence; fusing target covariables contained in the target covariable sequence with feature representations contained in the feature representation sequence based on an attention mechanism to obtain a feature representation sequence fused with the target covariables; and inputting the feature representation sequence fused with the target covariable into a prediction network, and performing time sequence prediction by the prediction network based on the feature representation sequence fused with the target covariable to obtain a prediction value of data contained in the time sequence in the target time period.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of artificial intelligence, and in particular, relate to a time series prediction method and a time series prediction model based on a time series prediction model. Background Art

[0002] A time series forecasting model is a mathematical model that analyzes historical observational data (time series) arranged in chronological order and then uses statistical methods, machine learning algorithms, or deep learning techniques to capture the temporal dependencies (such as trends, seasonality, cyclicality, and long-term dependencies) in the data. This model then infers the value or distribution of the target variable at a specific point in time or time period in the future. Its core goal is to predict unknown future values ​​based on the dynamic patterns of historical data.

[0003] In practical applications, when using a time series forecasting model to perform time series forecasting, in addition to the target variable directly related to the time series forecasting task, there may also be some external covariates. These covariates are important external information that helps the model understand the time series and often have an impact on the time series forecast results. In some forecasting scenarios where covariates play a key role, the impact of covariates on the forecast results may even be greater than that of the target variable directly related to the forecasting task.

[0004] Therefore, how to enable the time series prediction model to support external covariates in the process of using the time series prediction model to perform prediction tasks is of great significance for improving the prediction accuracy of the time series prediction model. Summary of the Invention

[0005] This specification proposes a time series prediction method based on a time series prediction model. The time series prediction model is a time series prediction model based on a deep learning network, including an encoding network and a prediction network. The encoding network is used to encode a time series into a feature representation sequence for time series prediction, and the prediction network is used to perform time series prediction based on the feature representation sequence. The method includes:

[0006] Obtain a time series corresponding to a historical period; and a target covariate sequence corresponding to a target period; wherein the target period is a future period after the historical period; the time series includes a plurality of data arranged in chronological order; the target covariate sequence includes a plurality of target covariates arranged in chronological order; the target covariate is a covariate that affects the prediction result of the data included in the time series in the target period;

[0007] Inputting the time series into the encoding network, and encoding the data contained in the time series into feature representations by the encoding network to obtain a feature representation sequence corresponding to the time series;

[0008] fusing the target covariates contained in the target covariate sequence with the feature representations contained in the feature representation sequence based on an attention mechanism to obtain a feature representation sequence fused with the target covariates;

[0009] The feature representation sequence fused with the target covariate is input into the prediction network, and the prediction network performs time series prediction based on the feature representation sequence fused with the covariate to obtain the predicted value of the data contained in the time series in the target time period.

[0010] Optionally, the time series prediction model further includes a preprocessing network located before the encoding network; wherein the preprocessing network is used to preprocess the data contained in the input time series to convert the data contained in the time series into a feature representation supported by the time series prediction model;

[0011] The time series is further input into the encoding network, comprising:

[0012] Inputting the time series into the preprocessing network, and having the preprocessing network preprocess the data contained in the time series respectively, so as to convert the data contained in the time series into a feature representation supported by the time series prediction model;

[0013] The time series after preprocessing is further input into the encoding network.

[0014] Optionally, before the time series is further input into the encoding network, the method further includes:

[0015] Obtaining a historical covariate sequence corresponding to the historical period; the historical covariate sequence includes a plurality of historical covariates arranged in chronological order; the historical covariates are covariates that affect actual values ​​of data included in the time series during the historical period;

[0016] fusing the historical covariates contained in the historical covariate sequence with the data contained in the time series based on an attention mechanism to obtain a time series fused with the historical covariates;

[0017] The time series fused with historical covariates is further input into the encoding network.

[0018] Optionally, the feature representation supported by the time series prediction model is a tokenized feature representation; the preprocessing network is used to perform tokenization processing on the data contained in the input time series to convert the data contained in the time series into a tokenized feature representation;

[0019] Inputting the time series into the preprocessing network, and having the preprocessing network preprocess the data contained in the time series respectively to convert the data contained in the time series into a feature representation supported by the time series prediction model, including:

[0020] The time series is input into the preprocessing network, and the preprocessing network performs tokenization processing on the data contained in the time series respectively to convert the data contained in the time series into tokenized feature representations.

[0021] Optionally, the covariate includes a heterogeneous covariate; wherein the heterogeneous covariate is a covariate that is heterogeneous with the data contained in the time series;

[0022] Before fusing the target covariates included in the target covariate sequence with the feature representations included in the feature representation sequence based on the attention mechanism, the method further includes:

[0023] converting heterogeneous covariates contained in the target covariate sequence into covariates that are isomorphic to the data contained in the time series;

[0024] Before fusing the historical covariates contained in the historical covariate sequence with the data contained in the time series based on the attention mechanism, the method further includes:

[0025] The heterogeneous covariates included in the historical covariate sequence are converted into covariates that are isomorphic to the data included in the time series.

[0026] Optionally, the data contained in the time series is numerical data;

[0027] Converting heterogeneous covariates into covariates that are homogeneous with the data contained in the time series includes:

[0028] Convert heterogeneous covariates into numerical data.

[0029] Optionally, the time series prediction model further includes a first fusion network located after the encoding network and before the prediction network; the target covariates corresponding to any time step contained in the target covariate sequence are composed of dynamic covariates and static covariates; the attention mechanism adopted by the first fusion network includes a conditional attention mechanism;

[0030] The target covariates included in the target covariate sequence are fused with the feature representations included in the feature representation sequence based on an attention mechanism, including:

[0031] Inputting the target covariate sequence into the first fusion network, so that the first fusion network uses the static covariates in the target covariates contained in the target covariate sequence as conditional information, and preliminarily fuses the dynamic covariates and the static covariates in the target covariates based on the conditional attention mechanism to obtain a covariate feature representation corresponding to the target covariate; and

[0032] Based on the self-attention mechanism, the covariate feature representation corresponding to the target covariate included in the target covariate sequence is further fused with the feature representation included in the feature representation sequence.

[0033] Optionally, the time series prediction model further includes a second fusion network located after the preprocessing network and before the encoding network; the attention mechanism adopted by the second fusion network includes a conditional attention mechanism; the historical covariates corresponding to any time step contained in the historical covariate sequence are composed of dynamic covariates and static covariates;

[0034] The historical covariates contained in the historical covariate sequence are fused with the data contained in the time series based on an attention mechanism, including:

[0035] Inputting the historical covariate sequence into the second fusion network, so that the second fusion network uses static covariates in the historical covariates contained in the historical covariates sequence as conditional information, and preliminarily fuses the dynamic covariates and static covariates in the historical covariates based on the conditional attention mechanism to obtain a historical covariate feature representation corresponding to the historical covariates;

[0036] The historical covariate feature representations corresponding to the historical covariates included in the historical covariate sequence are respectively added to the feature representations of the same time step included in the preprocessed time series.

[0037] Optionally, the first fusion network and the second fusion network include preset neural network units; wherein the neural network units are used to perform feature enhancement processing on the covariate feature representation;

[0038] Before further fusing the covariate feature representation corresponding to the target covariate included in the target covariate sequence with the feature representation included in the feature representation sequence based on the self-attention mechanism, the method further includes:

[0039] Inputting a covariate feature representation corresponding to a target covariate included in the target covariate sequence into the neural network unit, so that the neural network unit performs feature enhancement processing on the covariate feature representation;

[0040] Before adding the historical covariate feature representations corresponding to the historical covariates included in the historical covariate sequence to the feature representations of the same time step included in the preprocessed time series, the method further includes:

[0041] The historical covariate feature representation corresponding to the historical covariate included in the historical covariate sequence is input into the neural network unit, so that the neural network unit performs feature enhancement processing on the historical covariate feature representation.

[0042] Optionally, the neural network unit includes GLU.

[0043] Optionally, the time series prediction model includes a universal time series prediction model obtained by large-scale pre-training based on time series samples in multiple different fields.

[0044] Optionally, the time series prediction model includes a TSFM time series basic model.

[0045] Optionally, the training strategy used when training the time series prediction model includes:

[0046] Freeze the parameters of the preprocessing network, the encoding network, and the prediction network included in the network structure of the time series prediction model.

[0047] Optionally, the loss generated by training the time series prediction model includes:

[0048] The cumulative sum of the average quantile losses of each time step included in the target period; wherein the average quantile loss corresponding to any time step included in the target period is the average of the quantile losses of the time step at each quantile in the preset quantile set;

[0049] The quantile loss is used to describe the difference between the predicted value and the true value at that time step.

[0050] This specification also proposes a time series prediction model based on a deep learning network, including:

[0051] An encoding network is used to encode the data contained in the input time series corresponding to the historical period into a feature representation form to obtain a feature representation sequence corresponding to the time series; wherein the time series includes a plurality of data arranged in chronological order;

[0052] A prediction network is used to perform time series prediction based on an input feature representation sequence to obtain predicted values ​​of the data contained in the time series in the target time period; wherein the feature representation sequence input to the prediction network is a feature representation sequence fused with covariates obtained by fusing the target covariates contained in the target covariate sequence corresponding to the target time period with the feature representations contained in the feature representation sequence based on an attention mechanism; the target time period is a future time period after the historical time period; the target covariate sequence includes several target covariates arranged in chronological order; the target covariates are covariates that affect the prediction results of the data contained in the time series in the target time period.

[0053] Optionally, the time series prediction model further includes:

[0054] a preprocessing network located before the encoding network, configured to preprocess the data contained in the input time series corresponding to the historical period, so as to convert the data contained in the time series into a feature representation supported by the time series prediction model;

[0055] The encoding network is further used to encode the data contained in the time series input after preprocessing by the preprocessing network into a feature representation form to obtain a feature representation sequence corresponding to the time series.

[0056] Optionally, the target covariates corresponding to any time step contained in the target covariate sequence are composed of dynamic covariates and static covariates;

[0057] The time series prediction model also includes:

[0058] The first fusion network located after the encoding network and before the prediction network is used to take the static covariates in the target covariates contained in the input target covariate sequence corresponding to the target time period as conditional information, preliminarily fuse the dynamic covariates and static covariates in the target covariates based on the conditional attention mechanism, and obtain the covariate feature representation corresponding to the target covariate, and further fuse the covariate feature representation corresponding to the target covariate contained in the target covariate sequence with the feature representation contained in the feature representation sequence based on the self-attention mechanism.

[0059] Optionally, the historical covariates corresponding to any time step contained in the historical covariate sequence are composed of dynamic covariates and static covariates;

[0060] The time series prediction model also includes:

[0061] The second fusion network located after the preprocessing network and before the encoding network is used to take the static covariates in the historical covariates contained in the input historical covariate sequence corresponding to the historical period as conditional information, and preliminarily fuse the dynamic covariates and static covariates in the historical covariates based on the conditional attention mechanism to obtain the historical covariate feature representation corresponding to the historical covariate; and, add the historical covariate feature representation corresponding to the historical covariate contained in the historical covariate sequence to the feature representation of the same time step contained in the time series after preprocessing; wherein, the historical covariate sequence contains several historical covariates arranged in chronological order; the historical covariates are covariates that affect the actual values ​​of the data contained in the time series in the historical period.

[0062] In the above embodiment, by adopting the attention mechanism, the introduced covariates are fused with the feature representation for time series prediction obtained by encoding the data contained in the input time series by the encoding network of the time series prediction model. This can enable the time series prediction model to support external covariates in the process of time series prediction without destroying the original network structure of the time series prediction model and retaining the original prediction ability of the time series prediction model. This can not only enable the time series prediction model to adapt to more prediction scenarios where the covariates affect the time series prediction results, enhance the generalization ability of the time series prediction model, but also further improve the prediction accuracy of the time series prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0064] Figure 1 is a flowchart of a time series prediction method based on a time series prediction model shown in an embodiment of this specification;

[0065] Figure 2 is a network structure diagram of a time series prediction model shown in an embodiment of this specification;

[0066] Figure 3 is a network structure diagram of another time series prediction model shown in an embodiment of this specification;

[0067] Figure 4 is a schematic structural diagram of an electronic device shown in an embodiment of this specification;

[0068] Figure 5 This is a block diagram of a time series prediction device based on a time series prediction model shown in one embodiment of this specification. DETAILED DESCRIPTION

[0069] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments derived by those skilled in the art based on the embodiments in this specification without creative effort shall fall within the scope of protection of this specification.

[0070] In practical applications, when using time series prediction models to perform prediction tasks, enabling the time series prediction models to support external covariates is crucial to improving the accuracy of prediction results, especially in some prediction scenarios where covariates play a key role.

[0071] Currently, most time series prediction models are usually designed to make predictions based on a single variable and do not support external covariates. Therefore, when these time series prediction models are applied to prediction scenarios where covariates play a key role, the accuracy of time series predictions will be severely limited.

[0072] For example, taking the TSFMs (Time Series Foundation Models) system model as an example, TSFM is a general time series prediction model obtained through large-scale pre-training based on time series samples from multiple different fields. It aims to capture the complex structures and patterns in time series. By further fine-tuning the model, the model can be quickly adapted to various types of downstream tasks such as forecasting tasks.

[0073] TSFM typically uses a "channel-independence strategy" for pre-training. Based on this strategy, a multivariate time series can be split into multiple univariate time series. Each time series can be processed independently through different channels, while sharing the same model structure and parameters. In this way, each channel makes predictions based on a single variable and does not support external covariates.

[0074] With the continuous emergence of TSFMs, some types of TSFMs have adopted a covariate-aware strategy during pre-training to integrate external covariates to improve the model's ability to model complex time series patterns. However, while this strategy allows TSFM to support external covariates to a certain extent, it often destroys the model's original network structure and fails to preserve the model's original predictive capabilities. As a result, models trained in this way lack generalization and often struggle to achieve stable and excellent performance on a variety of tasks.

[0075] For example, based on this strategy, in order to enable the model to support external covariates, the time series of the input model is directly fused with the external covariates for joint modeling. This usually requires retraining the pre-trained model, and there is no guarantee that the parameters of the pre-trained model remain unchanged. Moreover, in order to adapt to this new requirement, the model architecture may also need to be redesigned.

[0076] Based on this, this specification proposes a general covariate adaptation framework that enables the time series prediction model to support external covariates without destroying the original network structure of the time series prediction model.

[0077] Based on this adaptation framework, the attention mechanism can be used on the basis of the original network structure of the time series prediction model to fuse the introduced covariates with the intermediate results generated by the time series prediction model when performing time series prediction. On the one hand, the original network structure of the time series prediction model can be preserved and the original prediction ability of the time series prediction model can be retained. On the other hand, the time series prediction model can be enabled to support external covariates during the time series prediction process.

[0078] In this way, the time series prediction model can eventually adapt to more prediction scenarios where covariates affect the time series prediction results. This can not only enhance the generalization ability of the time series prediction model, but also further improve the prediction accuracy of the time series prediction model.

[0079] See Figure 1 , Figure 1 This is a flowchart of a time series prediction method based on a time series prediction model shown in this specification; wherein the time series prediction model is a time series prediction model based on a deep learning network, including an encoding network and a prediction network; wherein the encoding network is used to encode the time series into a feature representation sequence for time series prediction, and the prediction network is used to perform time series prediction based on the feature representation sequence; the method includes the following execution process:

[0080] Step 102: Obtain a time series corresponding to a historical period and a target covariate sequence corresponding to a target period; wherein the target period is a future period after the historical period; the time series includes a plurality of data arranged in chronological order; the target covariate sequence includes a plurality of target covariates arranged in chronological order; the target covariates are covariates that affect the predicted values ​​of the data included in the time series in the target period;

[0081] The aforementioned covariates may include, in addition to the target variable directly related to the time series forecasting task, other variables that may affect the time series forecast results. These covariates help improve the accuracy of time series forecasting.

[0082] For example, in practical applications, typical covariates may include external events, economic indicators, interaction effects, etc. For example, when predicting sales in the future based on a time series consisting of sales data, variables such as promotional activities and weather conditions can be introduced as covariates.

[0083] The above-mentioned covariates generally include homogeneous covariates and heterogeneous covariates.

[0084] Homogeneous covariates are variables that have the same or similar characteristics as the variables contained in the time series used for time series forecasting. Assuming that the time series used for time series forecasting is composed of traditional numerical data (such as continuous real values), if the introduced covariates are also numerical data, then these covariates are usually called homogeneous covariates.

[0085] For example, suppose that when predicting the sales volume of ice cream in the future based on a time series composed of ice cream sales data, the temperature can be introduced as a covariate to assist in the prediction. Since the temperature, like the ice cream sales data, is numerical data, the temperature is a homogeneous covariate for the sales data.

[0086] A heterogeneous covariate is a variable that has characteristics that differ from those of the variables contained in the time series used for time series forecasting. Assuming that the time series used for time series forecasting is composed of traditional numerical data, if the covariate introduced is not numerical data, then this covariate is usually called a heterogeneous covariate.

[0087] For example, heterogeneous covariates usually include structured categorical variables (such as item IDs, calendar features) and multimodal inputs (such as images and text) in addition to numerical variables, etc.

[0088] It's also important to note that both homogeneous and heterogeneous covariates, in practical applications, can be composed of both dynamic and static covariates. Static covariates specifically refer to variables that remain constant throughout the time series; for example, geographic location, device type, user ID, etc. Dynamic covariates specifically refer to variables that change throughout the time series. For example, when predicting ice cream sales, the variables introduced are covariates that change dynamically.

[0089] The above-mentioned time series prediction model may specifically include any type of model for time series prediction built based on a deep learning network.

[0090] In some embodiments, the above-mentioned time series prediction model may specifically include a time series prediction model constructed based on a traditional neural network; for example, TiDE (Temporal Information-Driven Encoder-Decoder), TFT (Temporal Fusion Transformer) and other models.

[0091] In other embodiments, the time series prediction model may specifically include a general time series prediction model obtained by large-scale pre-training based on time series samples in multiple different fields. For example, in one example, the time series prediction model may specifically be a TSFMs series model.

[0092] In this specification, when performing time series forecasting based on the above-mentioned time series forecasting model, on the one hand, a time series corresponding to a historical period can be obtained; wherein the historical period can specifically refer to any time period in the past. The above-mentioned time series usually includes a number of data arranged in chronological order. For example, the time series can usually be a time series generated by arranging numerical data in chronological order. On the other hand, a target covariate sequence corresponding to a target period can also be obtained; wherein the above-mentioned target period specifically refers to a future period after the historical period; for example, assuming that the historical period is recorded as (1, T), the target period can specifically refer to (T, T+H). The above-mentioned target covariate sequence can specifically include a number of target covariates arranged in chronological order; the target covariate can specifically include a covariate that affects the predicted value of the data contained in the time series in the target period.

[0093] It should be noted that the above-mentioned process of performing time series prediction based on the above-mentioned time series prediction model can specifically be the time series prediction performed in the offline training stage of the time series prediction model, or it can be the online time series prediction based on the trained time series prediction model.

[0094] For example, in some embodiments, if the time series prediction based on the time series prediction model is performed during the offline training phase of the time series prediction model, the time series obtained at this time may specifically be a time series sample used as a training sample. In addition to including the time series data corresponding to the historical period, the time series sample may also include the actual value of the data in the time series during the target period. The actual value of the data in the time series during the target period may specifically serve as the sample label of the time series sample.

[0095] In other embodiments, if the time series prediction based on the above-mentioned time series prediction model is an online time series prediction based on a trained time series prediction model, the trained time series prediction model can be deployed online, and the above-mentioned time series obtained can specifically only include the time series data corresponding to the above-mentioned historical period.

[0096] Step 104: input the time series into the encoding network, and the encoding network encodes the data contained in the time series into feature representations to obtain a feature representation sequence corresponding to the time series;

[0097] See Figure 2 , Figure 2 This is a network structure diagram of a time series prediction model shown in this specification.

[0098] like Figure 2 As shown, in some embodiments, the above-mentioned time series prediction model may specifically include a preprocessing network, an encoding network and a prediction network.

[0099] The preprocessing network is located before the encoding network and is used to preprocess the data contained in the input time series to convert the data contained in the input time series into a feature representation supported by the time series prediction model.

[0100] For example, if the feature representation supported by a time series prediction model is tokenized, the preprocessing network can be used to tokenize the data contained in the input time series to convert it into a tokenized feature representation. In this case, the preprocessing network can act as a tokenizer.

[0101] It should be noted that the above-mentioned preprocessing network may be an optional network. In practical applications, if the data contained in the input time series itself is a feature representation supported by the time series prediction model, the network structure of the above-mentioned time series prediction model may not include the above-mentioned preprocessing network.

[0102] The encoding network is used to encode the input time series (such as the time series preprocessed by the preprocessing network) into a feature representation sequence for time series prediction.

[0103] The prediction network is used to perform time series prediction based on the feature representation sequence to obtain the predicted value of the data contained in the time series in the target time period.

[0104] It should be noted that the above-mentioned preprocessing network, the above-mentioned encoding network, and the above-mentioned prediction network are specifically for adapting the above-mentioned time series prediction model to the covariate adaptation framework proposed in this specification, and then further abstracting and decomposing the network structure of the time series prediction model according to the function of the time series prediction model to obtain network units. These networks are only logical networks, not physical networks. In actual applications, these networks can refer to a layer in the actual network structure of the time series prediction model, or a subnetwork composed of multiple layers in the actual network structure of the time series prediction model.

[0105] In this specification, after obtaining a time series corresponding to a historical period, the time series can be input into the above-mentioned encoding network, and the above-mentioned encoding network encodes the data contained in the time series into a feature representation form to obtain a feature representation sequence corresponding to the time series.

[0106] In some embodiments, see Figure 2 If the network structure diagram of the above-mentioned time series prediction model also includes a preprocessing network located before the encoding network, the acquired time series can be first input into the preprocessing network, and the preprocessing network will preprocess the data contained in the time series separately to convert the data contained in the time series into a feature representation supported by the time series prediction model.

[0107] For example, taking the feature representation supported by the time series prediction model as a tokenized feature representation, in this case, the preprocessing network can tokenize the data contained in the input time series and convert the data contained in the time series into a tokenized feature representation.

[0108] Then, the preprocessed time series is further input into the above encoding network, and the above encoding network encodes the data contained in the time series into the form of feature representation to obtain a feature representation sequence corresponding to the time series.

[0109] In some embodiments, before the preprocessed time series is further input into the encoding network, a historical covariate sequence corresponding to the historical period may also be obtained. Similar to the target covariate sequence, the historical covariate sequence may specifically include a number of historical covariates arranged in chronological order. The historical covariates may specifically include covariates that affect the actual values ​​of the data in the time series during the historical period. In other words, the historical covariates specifically refer to covariates that have already affected the data in the time series during the historical period.

[0110] After obtaining the historical covariate sequence corresponding to the above-mentioned historical period, the historical covariates contained in the historical covariate sequence can be fused with the data contained in the time series based on the attention mechanism to obtain a time series fused with historical covariates; then, the time series fused with historical covariates is further input into the above-mentioned encoding network, and the above-mentioned encoding network encodes the data contained in the time series into the form of feature representation to obtain a feature representation sequence corresponding to the time series.

[0111] In this way, a fusion mechanism for historical covariates can be introduced between the preprocessing network and the encoding network, and the attention mechanism can be used to fuse the preprocessed time series output by the preprocessing network with the historical covariates, so that the time series prediction model can also support historical covariates that have an impact on the actual values ​​of the data contained in the time series during the process of time series prediction for the time series.

[0112] In some embodiments, if there are heterogeneous covariates among the historical covariates contained in the above-mentioned historical covariate sequence, the heterogeneous covariates contained in the historical covariate sequence can be converted into covariates that are isomorphic with the data contained in the time series before the historical covariates contained in the historical covariate sequence are fused with the data contained in the time series based on the attention mechanism.

[0113] For example, in an example, taking the data contained in the above-mentioned time series as numerical data, when converting the heterogeneous covariates contained in the historical covariate sequence into covariates that are isomorphic to the data contained in the time series, the heterogeneous covariates can be specifically converted into numerical data.

[0114] Among them, it should be noted that when converting heterogeneous covariates into covariates that are isomorphic with the data contained in the time series, in addition to the need to convert the heterogeneous covariates into numerical data, in actual applications, it may also be necessary to convert the heterogeneous covariates into numerical data that is in the same numerical range as the data in the time series. The specific conversion method usually depends on the data specifications when the time series prediction model processes numerical data, and will not be described in detail in this manual.

[0115] Step 106: fusing the target covariates contained in the target covariate sequence with the feature representations contained in the feature representation sequence based on an attention mechanism to obtain a feature representation sequence that incorporates the target covariates.

[0116] In this specification, after the time series is input into the encoding network, and the encoding network encodes the data contained in the time series into the form of feature representation to obtain a feature representation sequence corresponding to the time series, the target covariates contained in the target covariate sequence corresponding to the target time period can be obtained based on the attention mechanism and fused with the feature representations contained in the feature representation sequence to obtain a feature representation sequence that integrates the target covariates.

[0117] In this way, a fusion mechanism for the target covariate can be introduced between the encoding network and the prediction network, and the attention mechanism can be used to fuse the feature representation sequence output by the encoding network with the target covariate, so that the time series prediction model can support the target covariate that affects the predicted value of the data contained in the time series in the future period during the process of time series prediction for the time series.

[0118] In some embodiments, if there are heterogeneous covariates among the target covariates contained in the above-mentioned target covariate sequence, the target covariates contained in the target covariate sequence corresponding to the target time period can be converted into covariates isomorphic to the data contained in the time series before the target covariates contained in the target covariate sequence corresponding to the target time period are obtained based on the attention mechanism and fused with the data contained in the time series.

[0119] For example, in one example, taking the data contained in the above-mentioned time series as numerical data, when converting the heterogeneous covariates contained in the target covariate sequence into covariates that are isomorphic to the data contained in the time series, the heterogeneous covariates can be specifically converted into numerical data.

[0120] In this specification, in order to avoid the two covariate fusion methods mentioned above that use the attention mechanism from destroying the network structure of the time series prediction model, a modular design can be adopted for the two covariate fusion methods mentioned above.

[0121] For example, the above-mentioned method of fusing the historical covariates contained in the historical covariate sequence with the data contained in the time series based on the attention mechanism can be inserted between the preprocessing network and the encoding network without interfering with the normal processing of the preprocessing network and the encoding network.

[0122] The above-mentioned method of fusing the target covariates contained in the target covariate sequence with the feature representations contained in the feature representation sequence output by the encoding network based on the attention mechanism can also be inserted between the encoding network and the prediction network without interfering with the normal processing of the encoding network and the prediction network.

[0123] It should be noted that when a modular design is adopted for the two covariate fusion methods mentioned above, the two covariate fusion methods mentioned above can be designed as an independent module or other execution unit (such as a component) that does not interfere with the normal processing of the existing network of the time series prediction model, and embedded between the preprocessing network and the encoding network or between the encoding network and the prediction network. The two covariate fusion methods mentioned above can also be designed as an independent network that does not interfere with the normal processing of the existing network of the time series prediction model, and embedded between the preprocessing network and the encoding network or between the encoding network and the prediction network. No specific limitation is made in this specification.

[0124] See Figure 3 , Figure 3 This is a network structure diagram of another time series prediction model shown in this specification.

[0125] In some embodiments, as Figure 3 As shown, in addition to further abstracting and decomposing the above-mentioned time series prediction model into a preprocessing network, an encoding network and the above-mentioned prediction network according to the function of the time series prediction model, an independent first fusion network can be designed between the encoding network and the prediction network, and an independent second fusion network can be designed between the preprocessing network and the encoding network.

[0126] Among them, the above-mentioned first fusion network (which can be called the post-fusion module) is specifically used to fuse the target covariates contained in the above-mentioned target covariate sequence with the feature representations contained in the feature representation sequence output by the encoding network based on the attention mechanism, so as to obtain a feature representation sequence that integrates the target covariates.

[0127] The second fusion network (which may be referred to as the pre-fusion module) is specifically used to fuse the historical covariates contained in the historical covariate sequence with the data contained in the preprocessed time series based on the attention mechanism to obtain a time series that incorporates the historical covariates.

[0128] In some embodiments, the target covariate sequence includes several target covariates arranged in chronological order, each target covariate corresponding to a time step. The target covariates corresponding to any time step in the target covariate sequence may specifically be composed of dynamic covariates and static covariates; that is, the target covariates corresponding to each time step may specifically be composed of two types of covariates, namely, dynamic covariates and static covariates. In this case, the attention mechanism adopted by the first fusion network may specifically include a conditional global attention mechanism.

[0129] Among them, the conditional attention mechanism is an attention mechanism adopted in the TFT (Temporal Fusion Transformer) model, which aims to enhance the model's ability to capture collective features across time series by combining the global attention mechanism with conditional information (such as external static features).

[0130] The core of the conditional attention mechanism lies in, on the one hand, using global attention, allowing the model to focus on the historical and future information of the entire time series when processing each time step, thereby capturing long-term dependencies and cross-sequence patterns. On the other hand, by introducing conditional information such as external static features, the attention weights are dynamically adjusted, enabling the model to focus on key features based on the context.

[0131] In this case, when the target covariates contained in the target covariate sequence are fused with the feature representations contained in the feature representation sequence based on the attention mechanism, the target covariate sequence can be input into a first fusion network, so that the first fusion network uses the static covariates in the target covariates contained in the target covariate sequence as conditional information, and preliminarily fuses the dynamic covariates and static covariates in the target covariates based on the conditional attention mechanism to obtain the covariate feature representation corresponding to the target covariate. Using the conditional attention mechanism, the attention weight of the dynamic covariate can be adjusted by introducing the static covariate as an external condition, thereby capturing the long-range dependency relationship between the static covariate and each target covariate contained in the target covariate sequence.

[0132] Among them, when the first fusion network preliminarily fuses the dynamic covariates and static covariates in the target covariate based on the conditional attention mechanism, it can specifically use the static covariate as conditional information as a parameter to calculate the attention weight corresponding to the dynamic covariate (usually calculated based on the Query and Key vectors), and then weight the attention weight with the calculated value vector of the dynamic covariate to obtain the covariate feature representation corresponding to the target covariate.

[0133] For example, the calculation process of preliminary fusion of dynamic covariates and static covariates in the target covariate based on the conditional attention mechanism can be expressed as follows:

[0134]

[0135] In the above formula, T:T+H represents the target period (T, T+H); S represents the static covariate; C T:T+H represents the dynamic covariate sequence consisting of the dynamic covariates of each time step in the target period (T, T+H); E S Represents the static covariate after vectorization; Represents the dynamic covariate sequence after vectorization; Represents a covariate feature representation sequence consisting of covariate feature representations corresponding to the dynamic covariates at each time step in the target period (T, T+H); CondGlobAttn represents the conditional global attention calculation. Since static covariates are all unchanged covariates, the conditional attention mechanism mentioned above is also called the conditional global attention mechanism; A represents the covariate feature representation of the target period (T, T+H). T:T+H The corresponding attention weight, V represents the T:T+H The corresponding Value vector; and Refers to S and C respectively T:T+H The corresponding attention weight matrix is ​​a learnable parameter matrix; P refers to C T:T+H The positions of the dynamic covariates in the dynamic covariate sequence are embedded; BMM refers to batch matrix multiplication, and softmax refers to the normalization function.

[0136] When the first fusion network takes the static covariates in the target covariates contained in the target covariate sequence as conditional information, and preliminarily fuses the dynamic covariates and static covariates in the target covariates based on the conditional attention mechanism to obtain the covariate feature representation corresponding to the target covariate, the covariate feature representation corresponding to the target covariate contained in the above target covariate sequence can be further fused with the feature representation contained in the above feature representation sequence output by the encoding network based on the self-attention mechanism to obtain a feature representation sequence that fuses the target covariate.

[0137] Among them, when the first fusion network further fuses the covariate feature representation corresponding to the target covariate contained in the above-mentioned target covariate sequence with the feature representation contained in the above-mentioned feature representation sequence output by the encoding network based on the self-attention mechanism, the above-mentioned target covariate sequence and the above-mentioned feature representation sequence can be spliced ​​as the input of the self-attention mechanism, and then the Key, Query and Value vectors of each feature representation contained in the spliced ​​sequence are calculated based on the self-attention mechanism, and then the attention weight corresponding to the feature representation is calculated based on the Key and Query vectors, and then the attention weight is weighted with the value vector of the feature representation to obtain the fused feature representation.

[0138] Assume that the target covariate sequence corresponds to the time period (T, T+H), and the feature representation sequence output by the encoding network corresponds to the time period (1, T). In this case, the self-attention mechanism ultimately outputs a feature representation sequence of length (1, T+H). Each time step of this feature representation sequence corresponds to a fused feature representation. In this case, the first fusion network can intercept the subsequence of the feature representation sequence within the time period (1, T) as the final output of the first fusion network, thereby obtaining a feature representation sequence that incorporates the target covariate.

[0139] In some embodiments, the historical covariate sequence also includes several historical covariates arranged in chronological order, each corresponding to a time step. The historical covariates corresponding to any time step in the historical covariate sequence may also specifically consist of dynamic covariates and static covariates; that is, the target covariate corresponding to each time step may specifically consist of two types of covariates: dynamic covariates and static covariates. In this case, the attention mechanism employed by the second fusion network may also specifically include a conditional attention mechanism.

[0140] In this case, when the historical covariates contained in the historical covariate sequence are fused with the data contained in the preprocessed time series based on the attention mechanism, the historical covariate sequence can be input into the second fusion network, so that the second fusion network uses the static covariates in the historical covariates contained in the historical covariate sequence as conditional information, and preliminarily fuses the dynamic covariates and the static covariates in the historical covariates based on the conditional attention mechanism to obtain a historical covariate feature representation corresponding to the historical covariate;

[0141] For example, the calculation process of preliminary fusion of dynamic covariates and static covariates in the target covariate based on the conditional attention mechanism can be expressed as follows:

[0142]

[0143] In the above formula, 1:T represents the historical period (1, T); C 1:T Represents the dynamic historical covariate sequence composed of the dynamic historical covariates of each time step in the historical period (1, T); the meanings of other parameters remain unchanged and will not be repeated here.

[0144] After the second fusion network preliminarily fuses the dynamic covariates and static covariates in the historical covariates to obtain the historical covariate feature representation corresponding to the historical covariates, the historical covariate feature representation corresponding to the historical covariates contained in the historical covariate sequence can be further fused with the feature representation contained in the preprocessed time series.

[0145] It should be noted that since the historical covariate sequence and the preprocessed time series are two sequences that are completely time-aligned and correspond to the same time period; therefore, when the second fusion network fuses the historical covariate feature representations corresponding to the historical covariates contained in the historical covariate sequence with the feature representations contained in the preprocessed time series, it is no longer necessary to use the self-attention mechanism for fusion. Instead, the historical covariate feature representations corresponding to the historical covariates contained in the historical covariate sequence can be added to the feature representations of the same time steps contained in the preprocessed time series to obtain the preprocessed time series that has been fused with the historical covariates.

[0146] For example, assuming that the time periods of the historical covariate sequence and the preprocessed time series are both (1, T), the historical covariate features corresponding to the historical covariates of the first time step in the historical covariate sequence can be added to the feature representation of the first time step in the preprocessed time series, and then the historical covariate features corresponding to the historical covariates of the second time step in the historical covariate sequence can be added to the feature representation of the second time step in the preprocessed time series, and so on, until the feature representations of all time steps are added.

[0147] It should be noted that in actual applications, if the historical covariate sequence corresponding to the above-mentioned historical period is not obtained, then the above-mentioned second fusion network may not be designed in the network structure of the above-mentioned time series prediction model, and only a first fusion network may be designed between the encoding network and the prediction network.

[0148] That is to say, if the historical covariate sequence corresponding to the above-mentioned historical period is not obtained, it is no longer necessary to fuse the historical covariates contained in the above-mentioned historical covariate sequence with the data contained in the time series after preprocessing, and only the target covariates contained in the above-mentioned target covariate sequence are fused with the feature representations contained in the feature representation sequence output by the encoding network.

[0149] In some embodiments, the first fusion network and the second fusion network may further include a preset neural network unit; wherein the neural network unit may be specifically used to perform feature enhancement processing on the covariate feature representation;

[0150] In this case, before the above-mentioned first fusion network further fuses the covariate feature representation corresponding to the target covariate contained in the above-mentioned target covariate sequence with the feature representation contained in the feature representation sequence output by the encoding network based on the self-attention mechanism, the covariate feature representation corresponding to the target covariate contained in the target covariate sequence can also be further input into the neural network unit, so that the neural network unit can perform feature enhancement processing on the covariate feature representation.

[0151] Similarly, before adding the historical covariate feature representations corresponding to the historical covariates contained in the above-mentioned historical covariate sequence and the feature representations of the same time steps contained in the preprocessed time series, the above-mentioned second fusion network can also input the historical covariate feature representations corresponding to the historical covariates contained in the historical covariate sequence into the neural network unit, so that the neural network unit can perform feature enhancement processing on the historical covariate feature representations.

[0152] In this way, the covariate features can be further enhanced before being fused, which helps to improve the accuracy of the model in performing time series forecasting.

[0153] Among them, the specific processing method of feature enhancement processing for covariate feature representation can be flexibly designed based on specific needs in practical applications; for example, the feature enhancement processing can specifically be to further extract key features from the covariate feature representation, and generate a new covariate feature representation based on the extracted key features.

[0154] In some embodiments, the above-mentioned neural network unit may specifically include a GLU (Gated Linear Unit); GLU is a neural network unit that introduces a gating mechanism and can perform feature enhancement processing on the input feature representation.

[0155] Step 108: Input the feature representation sequence integrated with the target covariate into the prediction network. The prediction network performs time series prediction based on the feature representation sequence integrated with the covariate to obtain the predicted value of the data contained in the time series in the target time period.

[0156] In this specification, after the target covariates contained in the target covariate sequence corresponding to the target time period are obtained based on the attention mechanism and fused with the feature representations contained in the feature representation sequence to obtain the feature representation sequence fused with the target covariates, the feature representation sequence fused with the target covariates can be further input into the above-mentioned prediction network, and the prediction network performs time series prediction based on the feature representation sequence fused with the target covariates to obtain the predicted value of the data contained in the time series in the target time period (that is, the future time period).

[0157] In this way, since the feature representation in the feature representation sequence output by the encoding network takes into account the impact of external covariates in the future period on the prediction results, the prediction accuracy of the time series prediction model can be improved.

[0158] It needs to be emphasized again that Figure 1 The process shown in steps 102 to 108 is intended to propose a mechanism for integrating external covariates with the input time series using an attention mechanism during time series prediction based on a time series prediction model. In practical applications, this mechanism can be integrated into both the offline training of the time series prediction model and the online time series prediction based on the trained time series prediction model.

[0159] That is to say, Figure 1The time series prediction based on the time series prediction model shown in steps 102 to 108 can be a time series prediction performed in the offline training stage of the time series prediction model, or an online time series prediction based on the trained time series prediction model.

[0160] The following will discuss these two situations separately.

[0161] In some embodiments, if Figure 1 The time series prediction based on the time series prediction model shown in steps 102 to 108 is a time series prediction performed in the offline training stage of the time series prediction model. At this time, the above-mentioned prediction network performs time series prediction based on the feature representation sequence that integrates the target covariate. After the data contained in the time series is predicted in the target time period (that is, the future time period), the training loss for the time series prediction model can be further calculated based on the predicted value.

[0162] Among them, in the scenario where a time series prediction model is used to perform prediction tasks, the training loss can be specifically constructed based on the difference between the predicted value of the data contained in the time series output by the above-mentioned prediction network in the target time period and the actual value of the data contained in the time series as the sample label in the target time period.

[0163] For example, in practical applications, the optimization objectives and model losses can be flexibly designed based on the difference between the predicted value and the actual value at each time step in the process of time series prediction for time series.

[0164] It should be noted that in the process of constructing the optimization objective and model loss, the specific type of loss function used is no longer specifically limited in this specification.

[0165] In some embodiments, a quantile loss function can be used to describe the difference between the predicted value and the actual value at a certain time step. Accordingly, when training the time series prediction model, the optimization objective can be to minimize the cumulative sum of the average quantile losses for each time step included in the target period.

[0166] In this case, the loss generated by training the time series prediction model may specifically include: the cumulative sum of the average quantile losses of each time step included in the target period.

[0167] Among them, the average quantile loss corresponding to any time step contained in the period can be specifically expressed by the average value of the quantile loss (Quantile Loss) of the time step at each quantile in the preset quantile set; and the quantile loss is usually used to describe the difference between the predicted value and the actual value at the time step.

[0168] For example, in this case, the loss function constructed for the time series prediction model based on the quantile loss function can be specifically expressed by the following formula;

[0169] In the above formula, q represents the quantile level, which represents the target quantile predicted by the model; represents a preset set of quantiles, Y τ Represents the predicted value at time step τ; represents the predicted value of quantile q at time step τ; H represents the total number of time steps contained in the time series; () + Defined as (x) + =max(0,x); It represents the cumulative sum of the average quantile losses of each time step contained in the above target period, which is the final output model loss.

[0170] In the above embodiment, quantile loss is specifically used to describe the difference between the predicted value and the actual value at a certain time step, and a loss function is constructed for the time series prediction model based on the quantile loss. It should be emphasized that in actual applications, other types of loss functions can also be used to describe the difference between the predicted value and the actual value at a certain time step, and based on the loss function, an optimization target is constructed for the time series prediction model.

[0171] For example, in some embodiments, a cross entropy loss function can also be used to describe the difference between the predicted value and the actual value at a certain time step, and an optimization target can be constructed for the time series prediction model based on the cross entropy loss function, which will not be listed one by one in this specification.

[0172] After further calculating the training loss for the time series prediction model, the model parameters included in the time series prediction model can be adjusted based on the training loss to complete a training process for the time series prediction model.

[0173] The specific process of adjusting the model parameters contained in the time series prediction model based on the calculated training loss will not be described in detail in this specification. Those skilled in the art can refer to the records in the relevant technology.

[0174] In some embodiments, the training strategy adopted when training the time series prediction model may specifically include a training strategy of freezing the parameters of the original preprocessing network, encoding network, and prediction network in the network structure of the time series prediction model.

[0175] For example, taking the above-mentioned time series prediction model as a general time series prediction model (such as the TSFMs series model) obtained by large-scale pre-training based on time series samples in multiple different fields, when further fine-tuning the pre-trained time series prediction model in a prediction scenario where covariates play a key role, the pre-trained parameters contained in the original preprocessing network, encoding network and prediction network in the network structure of the time series prediction model can be frozen, and only the parameters contained in the newly introduced first fusion network and second fusion network can be adjusted.

[0176] In this way, it can be ensured that when the above time series prediction model is further trained, the original network structure of the time series prediction model is not destroyed and the original prediction ability of the time series prediction model is retained.

[0177] Of course, in actual applications, the training strategy adopted when training the time series prediction model may include, in addition to the above-mentioned training strategy of freezing the parameters of the original preprocessing network, encoding network and prediction network in the network structure of the time series prediction model, other types of training strategies can also be flexibly introduced based on specific training goals and needs, which will not be listed one by one in this manual.

[0178] In some embodiments, if Figure 1 Steps 102 to 108 shown in the figure show the time series prediction based on the time series prediction model, and the online time series prediction based on the trained time series prediction model. At this time, the above-mentioned prediction network performs time series prediction based on the feature representation sequence that integrates the target covariate. The predicted value of the data contained in the time series in the target time period is the real prediction result predicted online by the time series prediction model.

[0179] In the above technical solution, by adopting the attention mechanism, the introduced historical covariate sequence corresponding to the historical time period is fused with the time series input to the time series prediction model, and the introduced target covariate sequence corresponding to the target time period is fused with the feature representation for time series prediction obtained by encoding the data contained in the input time series by the encoding network of the time series prediction model. This can enable the time series prediction model to support external covariates in the process of time series prediction without destroying the original network structure of the time series prediction model and retaining the original prediction ability of the time series prediction model. This can not only enable the time series prediction model to adapt to more prediction scenarios where covariates affect the time series prediction results, enhance the generalization ability of the time series prediction model, but also further improve the prediction accuracy of the time series prediction model.

[0180] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of an apparatus, an electronic device, and a storage medium.

[0181] Figure 4 This is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Figure 4 At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and may also include other required hardware. One or more embodiments of this specification can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0182] like Figure 5 As shown, Figure 5 This is a block diagram of a time series prediction device based on a time series prediction model according to an exemplary embodiment of the present specification. The device can be operated in Figure 4 The electronic device shown in the figure implements the technical solution of this specification. The time series prediction model is a time series prediction model based on a deep learning network, including an encoding network and a prediction network; wherein the encoding network is used to encode the time series into a feature representation sequence for time series prediction, and the prediction network is used to perform time series prediction based on the feature representation sequence; the device 50 includes:

[0183] Acquisition module 501, acquires a time series corresponding to a historical period; and a target covariate sequence corresponding to a target period; wherein the target period is a future period after the historical period; the time series includes a plurality of data arranged in chronological order; the target covariate sequence includes a plurality of target covariates arranged in chronological order; the target covariates are covariates that affect the predicted values ​​of the data included in the time series in the target period;

[0184] The encoding module 502 inputs the time series into the encoding network, and the encoding network encodes the data contained in the time series into a feature representation form to obtain a feature representation sequence corresponding to the time series;

[0185] A fusion module 503 is configured to fuse the target covariates contained in the target covariate sequence with the feature representations contained in the feature representation sequence based on an attention mechanism to obtain a feature representation sequence that incorporates the target covariates.

[0186] The prediction module 504 inputs the feature representation sequence integrated with the target covariate into the prediction network, and the prediction network performs time series prediction based on the feature representation sequence integrated with the covariate to obtain the predicted value of the data contained in the time series in the target time period.

[0187] Accordingly, this specification also provides an electronic device, which includes a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps in all the method flows described above.

[0188] Accordingly, this specification also provides a computer-readable storage medium on which executable computer program instructions are stored; wherein, when the instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0189] Accordingly, this specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0190] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0191] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0192] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude that with the future development of computer technology, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0193] Although one or more embodiments of this specification provide method operation steps as described in the embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any particular order.

[0194] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing one or more of the present specifications, the functions of each module can be implemented in the same or multiple software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0195] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0196] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0197] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0198] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0199] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0200] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0201] Those skilled in the art will appreciate that one or more embodiments of this specification may be provided as a method, system, or computer program product. Thus, one or more embodiments of this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0202] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0203] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between the various embodiments can be referenced across them. Each embodiment focuses on the differences from the other embodiments. In particular, since the system embodiments are generally similar to the method embodiments, their description is relatively simple. For relevant parts, reference can be made to the description of the method embodiments. Throughout this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and integrate the different embodiments or examples, and features of different embodiments or examples, described in this specification, without conflict.

[0204] The foregoing is merely an example of one or more embodiments of this specification and is not intended to limit the one or more embodiments of this specification. It will be apparent to those skilled in the art that various modifications and variations may be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this specification shall be included within the scope of the claims.

Claims

1. A time series prediction method based on a time series prediction model, wherein the time series prediction model is a time series prediction model based on a deep learning network, including an encoding network and a prediction network; wherein: The encoding network is used to encode the time series into a feature representation sequence for time series prediction, and the prediction network is used to perform time series prediction based on the feature representation sequence; the method includes: Obtain a time series corresponding to a historical period; and a target covariate sequence corresponding to a target period; wherein the target period is a future period after the historical period; the time series includes a plurality of data arranged in chronological order; the target covariate sequence includes a plurality of target covariates arranged in chronological order; the target covariate is a covariate that affects the predicted value of the data included in the time series in the target period; Inputting the time series into the encoding network, and encoding the data contained in the time series into feature representations by the encoding network to obtain a feature representation sequence corresponding to the time series; fusing the target covariates contained in the target covariate sequence with the feature representations contained in the feature representation sequence based on an attention mechanism to obtain a feature representation sequence fused with the target covariates; The feature representation sequence fused with the target covariate is input into the prediction network, and the prediction network performs time series prediction based on the feature representation sequence fused with the covariate to obtain the predicted value of the data contained in the time series in the target time period.

2. The method according to claim 1, wherein the time series prediction model further comprises a preprocessing network located before the encoding network; The preprocessing network is used to preprocess the data contained in the input time series to convert the data contained in the time series into a feature representation supported by the time series prediction model; The time series is further input into the encoding network, comprising: Inputting the time series into the preprocessing network, and having the preprocessing network preprocess the data contained in the time series respectively to convert the data contained in the time series into a feature representation supported by the time series prediction model; The time series after preprocessing is further input into the encoding network.

3. The method according to claim 2, further comprising: Obtaining a historical covariate sequence corresponding to the historical period; the historical covariate sequence includes a plurality of historical covariates arranged in chronological order; the historical covariates are covariates that affect actual values ​​of data included in the time series during the historical period; fusing the historical covariates contained in the historical covariate sequence with the data contained in the time series based on an attention mechanism to obtain a time series fused with the historical covariates; The time series fused with historical covariates is further input into the encoding network.

4. The method of claim 2, wherein the feature representation supported by the time series prediction model is a tokenized feature representation; and the preprocessing network is used to tokenize the data contained in the input time series to convert the data contained in the time series into the tokenized feature representation; Inputting the time series into the preprocessing network, and having the preprocessing network preprocess the data contained in the time series respectively to convert the data contained in the time series into a feature representation supported by the time series prediction model, including: The time series is input into the preprocessing network, and the preprocessing network performs tokenization processing on the data contained in the time series respectively to convert the data contained in the time series into tokenized feature representations.

5. The method of claim 3, wherein the covariates include heterogeneous covariates; The heterogeneous covariate is a covariate that is heterogeneous to the data contained in the time series; Before fusing the target covariates included in the target covariate sequence with the feature representations included in the feature representation sequence based on the attention mechanism, the method further includes: converting heterogeneous covariates contained in the target covariate sequence into covariates that are isomorphic to the data contained in the time series; Before fusing the historical covariates contained in the historical covariate sequence with the data contained in the time series based on the attention mechanism, the method further includes: The heterogeneous covariates included in the historical covariate sequence are converted into covariates that are isomorphic to the data included in the time series.

6. The method according to claim 5, wherein the data contained in the time series is numerical data; Converting heterogeneous covariates into covariates that are homogeneous with the data contained in the time series includes: Convert heterogeneous covariates into numerical data.

7. The method of claim 5, wherein the time series prediction model further comprises a first fusion network located after the encoding network and before the prediction network; the target covariates corresponding to any time step included in the target covariate sequence are composed of dynamic covariates and static covariates; and the attention mechanism employed by the first fusion network comprises a conditional attention mechanism; The target covariates included in the target covariate sequence are fused with the feature representations included in the feature representation sequence based on an attention mechanism, including: Inputting the target covariate sequence into the first fusion network, so that the first fusion network uses the static covariates in the target covariates contained in the target covariate sequence as conditional information, and preliminarily fuses the dynamic covariates and the static covariates in the target covariates based on the conditional attention mechanism to obtain a covariate feature representation corresponding to the target covariate; as well as, Based on the self-attention mechanism, the covariate feature representation corresponding to the target covariate included in the target covariate sequence is further fused with the feature representation included in the feature representation sequence.

8. The method of claim 7, wherein the time series prediction model further comprises a second fusion network located after the preprocessing network and before the encoding network; the attention mechanism employed by the second fusion network comprises a conditional attention mechanism; and the historical covariates corresponding to any time step contained in the historical covariate sequence comprise dynamic covariates and static covariates; The historical covariates contained in the historical covariate sequence are fused with the data contained in the time series based on an attention mechanism, including: Inputting the historical covariate sequence into the second fusion network, so that the second fusion network uses static covariates in the historical covariates contained in the historical covariates sequence as conditional information, and preliminarily fuses the dynamic covariates and static covariates in the historical covariates based on the conditional attention mechanism to obtain a historical covariate feature representation corresponding to the historical covariates; The historical covariate feature representations corresponding to the historical covariates included in the historical covariate sequence are respectively added to the feature representations of the same time step included in the preprocessed time series.

9. The method according to claim 8, wherein the first fusion network and the second fusion network include preset neural network units; The neural network unit is used to perform feature enhancement processing on the covariate feature representation; Before further fusing the covariate feature representation corresponding to the target covariate included in the target covariate sequence with the feature representation included in the feature representation sequence based on the self-attention mechanism, the method further includes: Inputting a covariate feature representation corresponding to a target covariate included in the target covariate sequence into the neural network unit, so that the neural network unit performs feature enhancement processing on the covariate feature representation; Before adding the historical covariate feature representations corresponding to the historical covariates included in the historical covariate sequence to the feature representations of the same time step included in the preprocessed time series, the method further includes: The historical covariate feature representation corresponding to the historical covariate included in the historical covariate sequence is input into the neural network unit, so that the neural network unit performs feature enhancement processing on the historical covariate feature representation.

10. The method of claim 9, wherein the neural network unit comprises a GLU.

11. The method according to claim 8, wherein the time series prediction model comprises a universal time series prediction model obtained by large-scale pre-training based on time series samples in multiple different fields.

12. The method according to claim 11, wherein the time series prediction model comprises a TSFM time series basic model.

13. The method of claim 11, wherein the training strategy employed when training the time series prediction model comprises: Freeze the parameters of the preprocessing network, the encoding network, and the prediction network included in the network structure of the time series prediction model.

14. The method according to claim 11, wherein the loss generated by training the time series prediction model comprises: The cumulative sum of the average quantile losses of each time step included in the target period; wherein the average quantile loss corresponding to any time step included in the target period is the average of the quantile losses of the time step at each quantile in the preset quantile set; the quantile loss is used to describe the difference between the predicted value and the actual value at the time step.

15. A time series prediction model based on a deep learning network, comprising: An encoding network is used to encode the data contained in the input time series corresponding to the historical period into a feature representation form to obtain a feature representation sequence corresponding to the time series; wherein the time series includes a plurality of data arranged in chronological order; A prediction network is used to perform time series prediction based on an input feature representation sequence to obtain predicted values ​​of the data contained in the time series in a target time period; wherein the feature representation sequence input to the prediction network is a feature representation sequence fused with covariates obtained by fusing the target covariates contained in the target covariate sequence corresponding to the target time period with the feature representations contained in the feature representation sequence based on an attention mechanism; the target time period is a future time period after the historical time period; the target covariate sequence includes several target covariates arranged in chronological order; the target covariates are covariates that affect the predicted values ​​of the data contained in the time series in the target time period.

16. The time series prediction model according to claim 15, further comprising: a preprocessing network located before the encoding network, configured to preprocess the data contained in the input time series corresponding to the historical period, so as to convert the data contained in the time series into a feature representation supported by the time series prediction model; The encoding network is further used to encode the data contained in the time series input after preprocessing by the preprocessing network into a feature representation form to obtain a feature representation sequence corresponding to the time series.

17. The time series prediction model according to claim 16, wherein the target covariates corresponding to any time step contained in the target covariate sequence are composed of dynamic covariates and static covariates; The time series prediction model also includes: The first fusion network located after the encoding network and before the prediction network is used to take the static covariates in the target covariates contained in the input target covariate sequence corresponding to the target time period as conditional information, preliminarily fuse the dynamic covariates and static covariates in the target covariates based on the conditional attention mechanism, and obtain the covariate feature representation corresponding to the target covariate, and further fuse the covariate feature representation corresponding to the target covariate contained in the target covariate sequence with the feature representation contained in the feature representation sequence based on the self-attention mechanism.

18. The time series prediction model according to claim 17, wherein the historical covariates corresponding to any time step contained in the historical covariate sequence are composed of dynamic covariates and static covariates; The time series prediction model also includes: The second fusion network located after the preprocessing network and before the encoding network is used to take the static covariates in the historical covariates contained in the input historical covariate sequence corresponding to the historical period as conditional information, and preliminarily fuse the dynamic covariates and static covariates in the historical covariates based on the conditional attention mechanism to obtain the historical covariate feature representation corresponding to the historical covariate; and, add the historical covariate feature representation corresponding to the historical covariate contained in the historical covariate sequence to the feature representation of the same time step contained in the time series after preprocessing; wherein, the historical covariate sequence contains several historical covariates arranged in chronological order; the historical covariates are covariates that affect the actual values ​​of the data contained in the time series in the historical period.

19. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 14 are implemented.

20. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 14.

Images