Time series prediction method and apparatus, electronic device, and medium

Abstract

The application discloses a time series prediction method and device, electronic equipment and medium, and relates to the technical field of big data. The method comprises the following steps: acquiring historical time series data; constructing a prediction model based on a Transformer network, wherein the prediction model comprises an attention layer, and the attention layer comprises a linear mapping autocorrelation attention module, which is used for acquiring autocorrelation features of input data of the prediction model; inputting the historical time series data into the prediction model as the input data, obtaining autocorrelation features of the historical time series data based on the linear mapping autocorrelation attention module of the prediction model, and obtaining a prediction sequence in a preset time period after a current time based on the autocorrelation features of the historical time series data. The method can capture context correlation information of the historical time series data, is more accurate in prediction, reduces spatial and temporal complexity, and improves memory efficiency and time efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular to a time series prediction method and device, electronic equipment and a medium. BACKGROUND

[0002] In mathematics, a time series is a series of data points indexed in time order, which is a sequence collected at continuous equidistant time points, and is a series of discrete time data. Time series prediction refers to predicting future values using a model based on previously observed values, and is a process of using statistical and modeling analysis of time series data to make predictions and provide information for strategic decision-making. Time series prediction can be used in any application science and engineering field involving time measurement, for example, it can be applied to statistics, signal processing, pattern recognition, econometrics, mathematical finance, weather forecasting, earthquake prediction, electroencephalogram, control engineering, astronomy, and communication engineering.

[0003] Currently, common time series prediction methods include machine learning-based time series prediction methods such as autoregressive moving average (ARMA), autoregressive (AR), etc. However, machine learning-based time series prediction methods are suitable for single-variable stationary time series analysis and are not suitable for multivariate time series prediction. Deep learning-based prediction methods such as recurrent neural network (RNN) and long short-term memory (LSTM) cannot completely eliminate the problems of gradient disappearance and gradient explosion when facing long time series, and each step of prediction depends on the hidden state of the previous step, which cannot fully represent the nonlinear relationship of time series data. SUMMARY

[0004] To solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present application provide a time series prediction method, device, electronic equipment and medium.

[0005] In a first aspect, the embodiments of the present application provide a time series prediction method, comprising:

[0006] obtaining historical time series data;

[0007] constructing a prediction model based on a Transformer network, the prediction model comprising an attention layer, the attention layer comprising a linear mapping autocorrelation attention module, the linear mapping autocorrelation attention module being configured to obtain autocorrelation features of input data of the prediction model;

[0008] input the historical time series data as input data into the prediction model, obtain self-correlation features of the historical time series data based on a linear mapping self-correlation attention module of the prediction model, and obtain a prediction sequence in a preset time period after a current time based on the self-correlation features of the historical time series data.

[0009] Optionally, the linear mapping self-correlation attention module of the prediction model comprises a linear mapping submodule and a self-correlation attention submodule; the linear mapping submodule is configured to perform linear mapping on a key matrix and / or a value matrix corresponding to the historical time series data based on a preset linear mapping matrix; and the self-correlation attention submodule is configured to perform self-correlation mapping on one or more of a query matrix, the key matrix and the value matrix corresponding to the historical time series data.

[0010] Optionally, the attention layer of the prediction model further comprises one or more of a sparse attention module, a residual attention module; the linear mapping self-correlation attention module, the sparse attention module, and the residual attention module are invoked by preset target parameters.

[0011] Optionally, the prediction model adopts an encoder-decoder structure; the encoder comprises an input embedding layer, a position encoding layer, the attention layer, a regularization layer and a feedforward layer; and the decoder comprises an output embedding layer, the position encoding layer, the attention layer, the regularization layer, the feedforward layer, a linear transformation layer and an activation layer.

[0012] Optionally, the position encoding layer of the prediction model is configured to determine a position encoding feature vector of each data in the historical time series data, and the position encoding layer adopts a trigonometric function coding.

[0013] Optionally, the position encoding layer determines the position encoding feature vector of each data in the historical time series data according to the following formula:

[0014]

[0015]

[0016] wherein, PE represents a position encoding feature vector, pos represents a position of data with an index subscript i in the historical time series data, N represents a positive integer, d model represents a model dimension of the prediction model.

[0017] Optionally, the feedforward layer of the prediction model comprises a first feedforward module and / or a second feedforward module, and the first feedforward module and the second feedforward module are invoked by preset specified parameters.

[0018] The activation function of the first feedforward module comprises one or more of the following: an ELU function, a GELU function, a GELU_Fast function, a GELU_new function, a Swish function, a Tanh function, and a Sigmoid function; and the second feedforward module is a convolutional neural network structure.

[0019] Optionally, the historical time series data is input as input data into the prediction model, and based on a linear mapping self-correlation attention module of the prediction model, a self-correlation feature of the historical time series data is obtained, including:

[0020] The historical time series data is input as input data into the prediction model, and the input embedding layer is used to convert the historical time series data into a vector form, so as to obtain an input data vector and a global time feature vector of the historical time series data;

[0021] The position encoding layer is used to determine a position encoding feature vector of the historical time series data;

[0022] Based on the input data vector, the global time feature vector, and the position encoding feature vector, an input representation vector is obtained.

[0023] The input representation vector is input into the linear mapping self-correlation attention module of the prediction model, and a self-correlation feature of the historical time series data is obtained.

[0024] Optionally, the decoder adopts a generative parallel prediction mode to obtain a prediction sequence in a preset time period after a current time.

[0025] The form of the input data of the decoder is shown in the following formula (2):

[0026]

[0027] wherein, represents a placeholder corresponding to the prediction sequence; represents a start character of the prediction sequence, and the start character is a time sequence sampled from the historical time series.

[0028] In a second aspect, an embodiment of the present application provides a time series prediction device, including:

[0029] A data acquisition module is configured to acquire historical time series data.

[0030] The model construction module is configured to construct a prediction model based on a Transformer network, the prediction model comprising an attention layer, the attention layer comprising a linear mapping self-correlation attention module configured to obtain a self-correlation feature of input data of the prediction model.

[0031] The prediction module is configured to input the historical time series data as input data into the prediction model, obtain a self-correlation feature of the historical time series data based on the linear mapping self-correlation attention module of the prediction model, and obtain a prediction sequence in a preset time period after the current time based on the self-correlation feature of the historical time series data.

[0032] Optionally, the prediction module is further configured to input the historical time series data as input data into the prediction model, convert the historical time series data into a vector form by using the input embedding layer to obtain an input data vector and a global time feature vector of the historical time series data, determine a position encoding feature vector of the historical time series data by using the position encoding layer, obtain an input representation vector based on the input data vector, the global time feature vector and the position encoding feature vector, and input the input representation vector into the linear mapping self-correlation attention module of the prediction model to obtain the self-correlation feature of the historical time series data.

[0033] In a third aspect, an electronic device is provided, which includes one or more processors and a memory device storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the time series prediction method according to any of the embodiments of the present application.

[0034] In a fourth aspect, a computer readable medium is provided, which stores a computer program, wherein when the program is executed by a processor, the time series prediction method according to any of the embodiments of the present application is implemented.

[0035] The time series prediction method according to the embodiments of the present application has the following advantages or beneficial effects:

[0036] The time series prediction method according to the embodiments of the present application can analyze and process historical time series data by using a prediction model constructed based on a Transformer network, and obtain a prediction sequence in a preset time period after the current time. The attention layer of the prediction model constructed based on the Transformer network comprises a linear mapping self-correlation attention module, which can obtain a self-correlation feature of historical time series data, capture context correlation information of the historical time series data, and predict more accurately. The complexity of the standard Transformer attention is reduced from O(N2) to O(N), and the prediction accuracy is improved.2 ) reduce to O(N), improve memory efficiency and time efficiency, so that the prediction model is more suitable for long sequence data.

[0037] The further effects of the above-mentioned non-conventional optional mode will be described in the following combined with specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings are used to better understand the present application, and do not constitute undue limitations on the present application. Among them:

[0039] Figure 1 The flowchart of the time series prediction method of the embodiment of the present application is shown;

[0040] Figure 2 The schematic diagram of the linear mapping autocorrelation attention module in the time series prediction method of the embodiment of the present application is shown;

[0041] Figure 3 The schematic diagram of the linear mapping autocorrelation attention module in the time series prediction method of the embodiment of the present application is shown;

[0042] Figure 4 The schematic diagram of the linear mapping autocorrelation attention module in the time series prediction method of another embodiment of the present application is shown;

[0043] Figure 5 The schematic diagram of the linear mapping autocorrelation attention module in the time series prediction method of another embodiment of the present application is shown;

[0044] Figure 6 The structural schematic diagram of the prediction model in the time series prediction method of the embodiment of the present application is shown;

[0045] Figure 7 The schematic diagram of the input representation vector of the time series prediction method of the embodiment of the present application is shown;

[0046] Figure 8 The tuning result schematic diagram of the prediction model of the embodiment of the present application is shown;

[0047] Figure 9 The structural schematic diagram of the time series prediction device of the embodiment of the present application is shown;

[0048] Figure 10 The structural schematic diagram of the electronic device of the embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, which are meant to be exemplary in nature, and in which specific details are set forth to provide an understanding of the present application. It will be apparent, however, to one skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the spirit and scope of the present application. Thus, the detailed description herein is to be regarded as merely illustrative in nature and is provided for the purpose of description and explanation only. Therefore, the description of the embodiments is not intended to limit the scope of the present application.

[0050] Figure 1 A flow chart of a time series prediction method of an embodiment of the present application is shown in FIG. 1, which includes the following steps: Figure 1

[0051] Step S101: Obtain historical time series data.

[0052] The historical time series data refers to a sequence of values of the same statistical indicator arranged in the order of their occurrence time. The historical time series data is a set of random variables sorted by time, which is usually the result of observing the same statistical indicator at a given sampling rate within equal time intervals. As an example, the historical time series data can be a sequence of values of the metropolitan network link flow monitoring indicator, the broadband online number or the order amount arranged in the order of their occurrence time. The prediction sequence can be the metropolitan network link flow monitoring indicator, the broadband online number or the order amount in a preset time period after the current time.

[0053] Step S102: Construct a prediction model based on a Transformer network, which includes an attention layer, and the attention layer includes a linear mapping autocorrelation attention module for obtaining autocorrelation features of the input data of the prediction model.

[0054] ​The standard Transformer network has poor effect in analyzing and predicting long time series data due to the time complexity and memory space complexity of the self-attention layer. To solve the technical problem, the prediction model improves the attention mechanism of the standard Transformer network, realizes a lightweight linear mapping self-correlation attention module, introduces a self-correlation attention mechanism, which can capture the context correlation information of the time sequence and provide long and short term dependence modeling, and predict more accurately. The self-correlation attention mechanism performs a self-correlation operation on the input data, which is a numerical differentiation technique. Calculating the autocorrelation of the time series helps to convert the non-stationary time series into a stationary form, which is used to eliminate the dependence of the sequence on time, and stabilize the mean value of the time series by eliminating the horizontal change of the time series. The self-correlation attention mechanism can use scaled dot-product attention to calculate the context mapping matrix, and then calculate the context embedding of each attention head. The above operation is approximated by a low-rank matrix, which reduces the complexity of the standard Transformer attention from O(N 2 ) to O(N) in time and space, while having higher memory efficiency and time efficiency.

[0055] Step S103: input the historical time series data as input data into the prediction model, obtain the autocorrelation features of the historical time series data based on the linear mapping self-correlation attention module of the prediction model, and obtain the prediction sequence in the preset time period after the current time based on the autocorrelation features of the historical time series data.

[0056] In this step, the historical time series data is input into the prediction model, and the autocorrelation features of the historical time series data are obtained through the linear mapping self-correlation attention module of the prediction model. The autocorrelation features are used to represent the context information of the historical time series data, and then the prediction model predicts the autocorrelation features of the historical time series data to obtain the prediction sequence in the preset time period after the current time.

[0057] The time series prediction method of the embodiment of the application analyzes and processes the historical time series data based on the prediction model constructed based on the Transformer network to obtain the prediction sequence in the preset time period after the current time. The attention layer of the prediction model constructed based on the Transformer network includes a linear mapping self-correlation attention module, which can obtain the autocorrelation features of the historical time series data, capture the context correlation information of the historical time series data, and predict more accurately. The complexity of the standard Transformer attention is reduced from O(N 2) is reduced to O(N), which improves the memory efficiency and time efficiency, and makes the prediction model more suitable for long sequence data.

[0058] In an optional embodiment, the linear mapping self-correlation attention module of the prediction model based on the Transformer network comprises a linear mapping sub-module and a self-correlation attention sub-module. The self-correlation attention sub-module is used to perform self-correlation mapping on one or more of the query matrix (Q matrix), the key matrix (K matrix), and the value matrix (V matrix) corresponding to the historical time series data.

[0059] The Q matrix, the K matrix, and the V matrix are linearly transformed from the input matrix X corresponding to the historical time series data, and the calculation formula is as follows:

[0060]

[0061] wherein W Q , W K , and W V are three trainable parameter matrices, and the input matrix X is multiplied by W Q , W K , and W V , respectively, to generate the Q matrix, the K matrix, and the V matrix through linear transformation.

[0062] In this embodiment, the self-correlation mapping of one or more of the Q matrix, the K matrix, and the V matrix by the self-correlation attention module comprises first-order self-correlation mapping of one or more of the Q matrix, the K matrix, and the V matrix and / or second-order self-correlation mapping of one or more of the Q matrix, the K matrix, and the V matrix. The first-order self-correlation is defined as: when the independent variable time series changes from t to t+1, the change Δy t of the time series feature value y=y(t) is called the first-order self-correlation of the time series feature function y(t) at time t:

[0063] Δy t =y(t+1)-y(t)=y t+1 -y t

[0064] For the input matrix X, the first-order self-correlation function is described as ACF(X).

[0065] The second-order self-correlation is defined as: when the independent variable time series changes from t to t+1, the self-correlation function of the first-order self-correlation:

[0066] Δ(Δy)=Δy t+1 -Δy t =(y t+2 -y t+1 )-(y t+1 -y t )

[0067] Δ(Δy) is called the second-order autocorrelation of the time-series characteristic function y(t) at time t. For the input matrix X, the second-order autocorrelation function is described as ACF. 2 (x).

[0068] Based on the definitions of first-order and second-order autocorrelation functions, the first-order autocorrelation attention mechanism of the V matrix is ​​used as an example for illustration, where the i-th attention head is:

[0069]

[0070] in, W i Q , Let E represent the weight matrix. i and F i Let E represent the linear projection transformation matrix. i ∈R dim_reduce_k×seq_ten , dim_reduce_k indicates that the dimension reduction matrix has dimension k, and dim_reduce_k*seq_len represents the constructed E i and F i Size, The size is seq_len × dim_reduce_k. The size is dim_reduce_k×d model .

[0071] Perform a first-order autocorrelation mapping on the V matrix:

[0072]

[0073] The linear mapping submodule is used to perform linear mapping on the key matrix and / or value matrix corresponding to historical time series data based on a preset linear mapping matrix.

[0074] Wherein, the linear mapping is a mapping from one vector space V to another vector space W and maintains the operation of addition and multiplication. In the embodiment, a linear mapping matrix is preset, and the Key matrix and / or the Value matrix are linearly mapped through the linear mapping matrix. As an example, an initial value of the linear mapping matrix can be generated from a normal distribution with a given mean and standard deviation, and a final linear mapping matrix is obtained through learning training. The embodiment of the application improves the matrix point multiplication operation of the standard Transformer network to linear operation in the matrix space through the linear mapping matrix, forms matrix decomposition of the attention of the linear mapping attention mechanism, and the random matrix formed can be approximated to a low-rank matrix, reducing the spatial complexity and time complexity of attention from O(N 2 ) to O(N), greatly reducing the consumption of memory and space, and having stronger memory and time utilization efficiency.

[0075] As an example, taking the first-order autocorrelation mapping of the V matrix as an example, as shown in Figure 2 , from the perspective of the structure of the algorithm, first, the V matrix is converted through the first-order autocorrelation mapping, two linear mapping matrices are added when calculating the K matrix and the V matrix, the scaled dot-product attention is used to calculate the context mapping matrix, and the context embedding of each attention head is calculated. The above operation only needs O(N) time and space complexity. As shown in Figure 3 , the Figure 3 first-order autocorrelation mapping is performed on the Q matrix, the K matrix and the V matrix from the perspective of the matrix.

[0076] Figure 4 is another variant of the autocorrelation attention mechanism of the embodiment of the application. In Figure 5 , first, the Q matrix is mapped through the second-order autocorrelation mapping, and the V matrix is mapped through the first-order autocorrelation mapping. Two linear projection matrices are added when calculating the K matrix and the V matrix, the scaled dot-product attention is used to calculate the context mapping matrix, and the context embedding of each attention head is calculated.

[0077] Figure 5 is another variant of the autocorrelation attention mechanism of the embodiment of the application. In Figure 5 , first, the Q matrix, the K matrix and the V matrix are all mapped through the first-order autocorrelation mapping. Two linear projection matrices are added when calculating the K matrix and the V matrix, the scaled dot-product attention is used to calculate the context mapping matrix, and the context embedding of each attention head is calculated. It should be noted that the above only describes the preferred embodiment of the application, and other variants are not described here.

[0078] In the optional embodiment, the attention layer of the prediction model constructed based on the Transformer network further includes one or more of the following: a sparse attention module, Attention modules and residual attention modules. Among them, linearly mapped autocorrelation attention modules, sparse attention modules, and... The attention module and residual attention module can be invoked using preset target parameters.

[0079] In implementing the time series prediction method of this invention, a qualitative analysis of the probability distribution of multi-head attention in a standard Transformer network reveals that only a small number of positions contribute significantly to attention, while most positions contribute little. Therefore, this invention constructs a sparse attention module, introducing sparse bias into attention calculation to reduce computational complexity and improve network performance. As an example, this sparse attention module can employ position-based sparse self-attention (atomic sparse mode, aggregated sparse attention, extended sparse attention), content-based sparse self-attention, etc. Furthermore, a sparse matrix of the same size as the query matrix is ​​introduced, containing only the Top K query vectors sampled for sparse evaluation. This allows self-attention to calculate only the inner product of the samples for each query key, reducing time and space complexity to O(NlogN). The sparse attention module in this embodiment adopts content-based sparse attention, where the temporal feature values ​​of its attention feature sampling are determined by the top K most similar feature values, controlling the time and space complexity to O(NlogN). It highlights the dominant features, enhances the robustness of the distillation operation, iteratively reduces the number of self-attention extraction layers, and concatenates the outputs of all encoder stacks to obtain the final latent vector representation of the encoder. The decoder generates prediction data for future time steps in parallel, rather than step by step, which greatly improves the inference speed of long sequence prediction and effectively captures the precise long dependencies between the output and the input.

[0080] Attention module borrows The method and the idea of ​​matrix decomposition, utilizing The method approximates the standard self-attention computation. The method is an effective technique for accelerating large-scale learning applications by generating low-rank approximations. Crucially, the method assumes that a matrix can be well approximated by processing only a subset of it. Standard Transformer models use linear attention to reduce attention complexity through the associative law of matrix multiplication. The prediction model based on a Transformer network in this invention reduces the complexity of attention by processing only a subset of the matrix. n×d Clustering transformation forms a new matrix pass The Attention attention matrix is constructed in a double Softmax form, and a linear approximation attention closer to the standard attention is gradually searched, and experiments show that, The approximation attention module is competitive compared with the standard self-attention module.

[0081] Residual attention module: the prediction model constructed based on the Transformer network in the embodiment of the application constructs a residual attention module. Compared with the Pre-LN (Pre Layer Normalization) and Post-LN (Post Layer Normalization) of the standard Transformer, the residual attention module adds a Residual connection between the attention modules, which not only maintains the performance of the Post-LN, but also integrates the Residual connection. Experiments show that the residual attention module is superior to the standard Transformer framework.

[0082] In an optional embodiment, the prediction model constructed based on the Transformer network adopts an encoder-decoder structure.

[0083] As shown in Figure 6 , the encoder of the prediction model includes an input embedding layer, a position encoding layer, the attention layer, a regularization layer and a feedforward layer. The decoder of the prediction model includes an output embedding layer, the position encoding layer, the attention layer, the regularization layer, the feedforward layer, a linear transformation layer and a Softmax layer.

[0084] The input embedding layer is used to convert the historical time series data into a vector form.

[0085] The position encoding layer is used to determine the position encoding feature vector of each data in the historical time series data, and the position encoding layer adopts a trigonometric function coding. Through the position encoding layer, the expression ability of the absolute position information and the relative position information is obtained, and the relative or absolute position information of the input sequence is injected.

[0086] In order to handle the sequence problem, the solution of the standard Transformer network is to use position encoding (Position Encode / Embedding, PE), which maps a fixed position vector to each position in the sequence, and then adds the embedded word vector and the position vector to obtain the final input vector of each time sequence as the input of the bottom of the encoder and decoder stack. The position encoding layer of the embodiment of the application uses a trigonometric function position encoding, each dimension of the position encoding corresponds to a sine curve, the wavelength forms a geometric progression from 2π to N·2π (N is a positive integer, for example, 10000), and the position encoding is realized by using the sine and cosine functions. Specifically, the position encoding layer determines the position encoding feature vector by the following formula:

[0087]

[0088] Wherein, PE represents the position encoding feature vector, pos represents the position of the data with index i in the historical time series data, N represents a positive integer, d model represents the model dimension of the prediction model.

[0089] The attention layer includes the linear mapping self-correlation attention module described in the above embodiment, and can also include a sparse attention module, one or more of the attention module and the residual attention module.

[0090] The update of each weight parameter in the deep neural network will cause the input distribution of each layer to change. In order to solve the data distribution deviation problem, the standard Transformer network introduces a regularization layer (normalization layer) to keep the input distribution of the neural layer consistent during training. Unlike batch normalization (Batch Normalization), the regularization layer of the prediction model of the embodiment of the application realizes layer normalization (Layer Normalization), which summarizes the data of the same layer, calculates the mean and variance, and normalizes the input data of each layer to speed up the convergence of the deep network.

[0091] In the optional embodiment, the step S103 inputs the historical time series data as input data into the prediction model, and obtains the autocorrelation features of the historical time series data based on the linear mapping self-correlation attention module of the prediction model, including:

[0092] The historical time series data is input as input data into the prediction model, and the input embedding layer is used to convert the historical time series data into a vector form to obtain an input data vector and a global time feature vector of the historical time series data.

[0093] The location-encoded feature vector of historical time series data is determined using a location-encoded layer.

[0094] Based on the input data vector, global temporal feature vector, and positional encoding feature vector, the input representation vector is obtained;

[0095] The input representation vector is input into the linear mapping autocorrelation attention module of the prediction model to obtain the autocorrelation features of historical time series data.

[0096] The historical time series data obtained in step S101 is data without time series information. To enable the prediction model to analyze this historical time series data, this embodiment of the invention provides a unified input representation, such as... Figure 7 As shown, features from local time stamp, global time stamp, and value embedding are integrated to form a new input representation vector. The input data vector is obtained by encoding historical time series data using a preset encoding method. The global time stamp is obtained by encoding the corresponding time of the historical time series data, and the location encoding feature vector is obtained by encoding the local position of the historical time series data. Local location encoding injects the sequential position information of the marker in the sequence, embedding temporal context information at fixed positions. In this embodiment, the location encoding layer uses sine and cosine functions to implement location encoding, using sine encoding for even positions and cosine encoding for odd positions, injecting the local location encoding information of the sequence into the input sequence, enabling the prediction model of this embodiment to learn temporal information. Global time-series encoding injects global time features (minute, hour, day, week, month, year) of the marked sequence. The prediction model embeds local location features and global time features into the input vector at the bottom of the encoder and decoder stacks. After the above steps, the input representation vector of the encoder model is obtained.

[0097]

[0098] Where i∈{1,…,L x}, a is a factor of the size between the input data vector, the local position encoding feature vector and the global time feature vector, VE represents the input data vector (Value Embedding), as an example, a = 1, PE represents the position encoding feature vector (Local Time Stamp), GE represents the global time feature vector (Global TimeStamp), n represents the global time feature vector including n types, such as minute (minute), hour (hour), day (day), week (week), month (month), year (year).

[0099] In an optional embodiment, the feedforward layer of the prediction model comprises a first feedforward module and / or a second feedforward module, and the first feedforward module and the second feedforward module are called by preset specified parameters. The activation function of the first feedforward module comprises one or more of the following: ELU function, GELU function, GELU_Fast function, GELU_new function, Swish function, Tanh function, Sigmoid function; the second feedforward module is a convolutional neural network structure.

[0100] In the structure of the standard Transformer network, simply stacking attention modules will cause the problem of layer collapse and uniform inductive bias of labels, and the construction of the feedforward layer (FFN) in the embodiment of the application can alleviate this problem. The prediction module of the embodiment of the application comprises a first feedforward module and / or a second feedforward module, and the first feedforward module and the second feedforward module are called by preset specified parameters. In an optional embodiment, the prediction model constructed based on the Transformer network can also remove the feedforward layer, thereby simplifying the network.

[0101] The second feedforward module is a convolutional neural network structure, wherein the convolutional neural network is composed of neurons with learnable weights and biases, each neuron receives sequence input information, a convolutional neural network layer stacks multiple filters together, and a convolutional output result is obtained through two one-dimensional convolution and Dropout operations. With the deepening of the convolutional neural network, more complex global features are combined. Compared with the FFN network, the implementation efficiency is higher, and the number of parameters in the network is reduced. Dropout means that during model training, the weights of some hidden layer nodes in the network are randomly made not to work, the nodes that do not work are temporarily considered not to be part of the network structure, but the weights are retained (only temporarily not updated), and the next sample input may work again.

[0102] Based on the above prediction model, the process of the time series prediction method of the embodiment of the application can include input vector representation, encoder data conversion, decoder decoding and the like. Overall, a sliding window of a given size is preset, and at time t, the input vector The prediction model outputs a corresponding prediction sequence Wherein the input and output feature dimensions can include multiple features.

[0103] Input vector representation process: historical time series data is data without time sequence information, in order to enable the prediction model to analyze the historical time series data, the embodiment of the application gives a unified input representation, as shown in Figure 7 The local position encoding (Local Time Stamp), global time feature vector (Global Time Stamp), and value embedding feature of the input data vector are combined to form a new input representation vector. The local position encoding marks the order position information in the sequence, and embeds the time sequence context information through fixed position. In the embodiment of the application, the position encoding is realized by using the sine function, the sine encoding is used in the even bits, and the cosine encoding is used in the odd bits. The local position encoding information of the sequence is injected into the input sequence, so that the prediction model of the embodiment of the application has the ability to learn time sequence information. The global level time encoding marks the global time features (minute, hour, day, week, month, year) of the sequence. The prediction model embeds the local position feature and the global time feature in the input vector at the bottom of the encoder and decoder stack. After the above steps, the input representation vector of the encoder model is obtained:

[0104]

[0105] Wherein, i∈{1,…,L x}, α is a factor balancing the size between the input data vector, the local position encoding feature vector and the global time feature vector, VE represents the input data vector (Value Embedding), as an example, α = 1, PE represents the position encoding feature vector (Local Time Stamp), GE represents the global time feature vector (Global Time Stamp), and n represents the global time feature vector including n types, such as minute (minute), hour (hour), day (day), week (week), month (month), and year (year).

[0106] Encoder data conversion: the TeleTransformer encoder extracts the dependency relationship of long sequence data, and the tth sequence feed-in data is represented as a matrix:

[0107]

[0108] The attention layer of the prediction model of the embodiment of the application adopts a linear mapping self-correlation attention module, which is based on the attention mechanism of first-order and second-order self-correlation, such as the first-order self-attention head of a V vector:

[0109]

[0110] wherein, W i Q 、 represents a weight matrix, E i and F i represent linear projection conversion matrices, E i ∈R dim_reduce_k×seq_ten , dim_reduce_k represents a dimension reduction matrix with a dimension of k, and dim_reduce_k*seq_len represents the size of E i and F i constructed, the size of E model is seq_len*dim_reduce_k, the size of F model is dim_reduce_k*d model .

[0111] The linear mapping self-correlation attention module of the embodiment of the application is approximated by a low-rank matrix, which reduces the complexity of the standard Transformer attention from O(N 2 ) to O(N) in time and space, and has higher memory and time efficiency.

[0112] Decoder data conversion process: the decoder of the prediction model of the embodiment of the application uses two Multi-HeadAttention layers stacked to form a decoder structure, and the following vectors are fed into the decoder of the prediction model:

[0113]

[0114] wherein, represents a start mark of a prediction sequence, as shown in Figure 7 , an L token sequence is sampled in the input sequence, and represents the historical input information before the output prediction sequence, where token represents the feature value of each time step in the input sequence, for example, the number of broadband users of each time sequence, and 0 represents that the value of the future prediction time step is preset to 0. As a specific example, when the historical time sequence data is the number of broadband online and the number of orders observed in the past period of time (for example, in the past 80 hours), the start mark of the prediction sequence can be the number of broadband online and the number of orders observed in the recent hours (for example, in the recent 8 hours), and the prediction sequence can be the number of broadband online and the number of orders in the future hours (for example, in the future 5 hours).

[0115] The traditional Seq2Seq model decoding method includes a search-based decoding method (greedy search, beam search) and a sampling-based decoding method (random sampling, Topk sampling). The greedy search decoding method only predicts one value at each time step, and generates the value of the second time step according to the result of the encoder and the value of the first time step, then generates the value of the third time step according to the result of the encoder and the values of the first two time steps, and so on. This method cannot obtain the global optimum. The prediction model of the embodiment of the application adopts a parallel prediction method, realizes parallel training of the decoder, inputs the feature values of the entire input time sequence to the decoder, and parallelly calculates n prediction values based on the mask self-attention algorithm, which correspond to the outputs of n time points respectively. Only one step is needed to realize the prediction of all results. In the prediction model of the embodiment of the application, the mask in the Masked Multi-Head Attention is set to negative infinity (-∞), so that the ProbSparse Self-Attention target sequence prediction prevents each position from paying attention to the future position. When predicting, only the historical input information can be used for prediction, and the autoregression when predicting is avoided. The MAE (Mean Absolute Error) average absolute error is selected as the loss function according to the prediction of the target sequence, and the output of the decoder is back propagated to the entire model.

[0116] In an optional embodiment, after the prediction model is constructed, the prediction model can be subjected to hyperparameter optimization.

[0117] The hyperparameter optimization is mainly implemented from two aspects: network design related parameters: the number of encoder-decoder network layers, the number of attention heads, the parameter setting of hidden layer neurons, and the sequence length parameter; and model training process related parameters: the size of small batch data, the learning rate, the number of iterations, and the like. The hyperparameter optimization experimental results of the embodiment of the application are as follows: Figure 8And as shown in Table 1 below, it can be seen from the experiment that: the model MAE and MSE indicators are very sensitive to the number of attention heads and the size of the model dimension; not only the number of encoder-decoder network layers, but also the size of the small batch data is very important. In the embodiment, the best practice of adjusting the TeleTransformer parameters is used to improve the generalization ability of the model and accelerate the convergence speed of the model training.

[0118] Table 1:

[0119]

[0120]

[0121] Figure 9 A structural schematic diagram of a time series prediction device 900 according to an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the time series prediction device 900 includes: Figure 9 A data acquisition module 901 configured to acquire historical time series data.

[0122] A model construction module 902 configured to construct a prediction model based on a Transformer network, the prediction model including an attention layer, the attention layer including a linear mapping self-correlation attention module configured to acquire self-correlation features of input data of the prediction model.

[0123] A prediction module 903 configured to input the historical time series data as input data into the prediction model, acquire self-correlation features of the historical time series data based on the linear mapping self-correlation attention module of the prediction model, and obtain a prediction sequence in a preset time period after a current time based on the self-correlation features of the historical time series data.

[0124] In an optional embodiment, the prediction module is further configured to input the historical time series data as input data into the prediction model, convert the historical time series data into a vector form by using an input embedding layer of the prediction model to obtain an input data vector and a global time feature vector of the historical time series data; determine a position encoding feature vector of the historical time series data by using a position encoding layer of the prediction model; acquire an input representation vector based on the input data vector, the global time feature vector, and the position encoding feature vector; input the input representation vector into the linear mapping self-correlation attention module of the prediction model to obtain the self-correlation features of the historical time series data.

[0125]

[0126] ​The apparatus can execute the method provided by the embodiments of the application, and has the corresponding function modules and beneficial effects of executing the method. Technical details not described in the embodiments can be referred to the method provided by the embodiments of the application.

[0127] The embodiments of the application also provide an electronic device, such as Figure 10 As shown in the figure, the electronic device includes a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002 and the memory 1003 complete mutual communication through the communication bus 1004,

[0128] The memory 1003 is used for storing a computer program.

[0129] The processor 1001 is used for executing the program stored in the memory 1003, and the following steps are implemented:

[0130] Obtaining historical time series data;

[0131] Constructing a prediction model based on a Transformer network, the prediction model including an attention layer, the attention layer including a linear mapping autocorrelation attention module, the linear mapping autocorrelation attention module being used for obtaining autocorrelation features of input data of the prediction model;

[0132] Inputting the historical time series data as input data into the prediction model, obtaining autocorrelation features of the historical time series data based on the linear mapping autocorrelation attention module of the prediction model, and obtaining a prediction sequence in a preset time period after a current time based on the autocorrelation features of the historical time series data.

[0133] The communication bus 1004 mentioned above can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 1004 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0134] The communication interface 1002 is used for communication between the terminal and other devices.

[0135] The memory 1003 can include a random access memory (RAM) and can also include a non-volatile memory, such as at least one disk storage. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor 1001.

[0136] The processor 1001 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0137] In yet another embodiment provided by the present application, a computer readable medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform the time series prediction method of any of the above embodiments.

[0138] In yet another embodiment provided by the present application, a computer program product containing instructions is also provided, which, when executed on a computer, causes the computer to perform the time series prediction method of any of the above embodiments.

[0139] In the embodiments described above, all or some of the steps can be implemented by software, hardware, firmware or any combination thereof. When implemented in software, all or some of the steps can be implemented in the form of one or more computer programs which are stored in a computer readable storage medium. The computer readable storage medium can be located in a computing device which is in operation. These computer programs (which may

[0140] It should be noted that, in the present document, the terms such as first and second are used only to differentiate one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Also, the terms "comprising", "containing" or any other similar term are intended to encompass non-exclusive inclusion such that a process, method, article or apparatus that comprises a list of elements does not necessarily include only those elements in the list, but can include other elements not expressly listed or inherent to such process, method, article or apparatus.

[0141] Each of the embodiments in the present document is described in a related manner, and the same or similar parts among the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments.

[0142] The above merely provides the preferred embodiments of the application, and not intended to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall fall within the protection scope of the application.

Claims

1. A time series prediction method characterized by, The method comprises: obtaining historical time series data; the historical time series data is a sequence arranged in chronological order according to a metropolitan area network link traffic monitoring index and a broadband online number; constructing a prediction model based on a Transformer network, the prediction model comprising an attention layer, the attention layer comprising a linear mapping self-correlation attention module, the linear mapping self-correlation attention module being configured to obtain a self-correlation feature of input data of the prediction model; inputting the historical time series data as input data into the prediction model, obtaining a self-correlation feature of the historical time series data based on the linear mapping self-correlation attention module of the prediction model, and obtaining a prediction sequence in a preset time period after a current time based on the self-correlation feature of the historical time series data; the prediction sequence is the metropolitan area network link traffic monitoring index and the broadband online number in the preset time period after the current time; the linear mapping self-correlation attention module of the prediction model comprises a linear mapping submodule and a self-correlation attention submodule; the linear mapping submodule is configured to perform linear mapping on a key matrix and / or a value matrix corresponding to the historical time series data based on a preset linear mapping matrix; and the self-correlation attention submodule is configured to perform self-correlation mapping on one or more of a query matrix, the key matrix, and the value matrix corresponding to the historical time series data; inputting the historical time series data as input data into the prediction model, converting the historical time series data into a vector form by using an input embedding layer to obtain an input data vector and a global time feature vector of the historical time series data; determining a position encoding feature vector of the historical time series data by using a position encoding layer; obtaining an input representation vector based on the input data vector, the global time feature vector, and the position encoding feature vector; and inputting the input representation vector into the linear mapping self-correlation attention module of the prediction model to obtain a self-correlation feature of the historical time series data.

2. The method of claim 1, wherein, The attention layer of the prediction model further comprises one or more of a sparse attention module, a Nyström attention module, and a residual attention module; the linear mapping self-correlation attention module, the sparse attention module, the Nyström attention module, and the residual attention module are called by using preset target parameters.

3. The method of claim 2, wherein, The prediction model adopts an encoder-decoder structure. The encoder comprises an input embedding layer, a position encoding layer, the attention layer, a regularization layer, and a feedforward layer; and the decoder comprises an output embedding layer, the position encoding layer, the attention layer, the regularization layer, the feedforward layer, a linear transformation layer, and an activation layer.

4. The method of claim 3, wherein, The position encoding layer of the prediction model is configured to determine a position encoding feature vector of each data in the historical time series data, and the position encoding layer adopts a trigonometric function coding.

5. The method of claim 4, wherein, The position encoding layer determines the position encoding feature vector of each data in the historical time series data according to the following formula (1): wherein, represents a position encoding feature vector, pos represents a position of data with index subscript i in the historical time series data, N represents a positive integer, represents a model dimension of the prediction model.

6. The method of claim 4, wherein, The feedforward layer of the prediction model comprises a first feedforward module and / or a second feedforward module, and the first feedforward module and the second feedforward module are called by preset specified parameters; The activation function of the first feedforward module comprises one or more of the following: an ELU function, a GELU function, a GELU_Fast function, a GELU_new function, a Swish function, a Tanh function, and a Sigmoid function; The second feedforward module is a convolutional neural network structure.

7. The method of claim 6, wherein, The historical time series data is input as input data into the prediction model, and based on a linear mapping self-correlation attention module of the prediction model, a self-correlation feature of the historical time series data is obtained, comprising: The historical time series data is input as input data into the prediction model, and the input embedding layer is used to convert the historical time series data into a vector form, to obtain an input data vector and a global time feature vector of the historical time series data; The position encoding layer is used to determine a position encoding feature vector of the historical time series data; Based on the input data vector, the global time feature vector and the position encoding feature vector, an input representation vector is obtained; The input representation vector is input into the linear mapping self-correlation attention module of the prediction model, and a self-correlation feature of the historical time series data is obtained.

8. The method of claim 7, wherein, The decoder adopts a generative parallel prediction mode to obtain a prediction sequence in a preset time period after the current time; The form of the input data of the decoder is shown in the following formula (2): （2） wherein, represents a placeholder corresponding to the predicted sequence; represents a start character of the predicted sequence, the start character being a time series sampled from the historical time series.

9. A time series prediction device characterized by comprising: Comprise: The data acquisition module is configured to acquire historical time series data, wherein the historical time series data is a sequence arranged in chronological order according to a metropolitan area network link flow monitoring index and a broadband online number; The model construction module is configured to construct a prediction model based on a Transformer network, wherein the prediction model comprises an attention layer, the attention layer comprises a linear mapping self-correlation attention module, and the linear mapping self-correlation attention module is configured to acquire a self-correlation feature of input data of the prediction model; The prediction module is configured to input the historical time series data as input data into the prediction model, to acquire a self-correlation feature of the historical time series data based on a linear mapping self-correlation attention module of the prediction model, and to acquire a prediction sequence in a preset time period after the current time based on the self-correlation feature of the historical time series data, wherein the prediction sequence is the metropolitan area network link flow monitoring index and the broadband online number in the preset time period after the current time; The linear mapping self-correlation attention module of the prediction model comprises a linear mapping submodule and a self-correlation attention submodule; The linear mapping submodule is configured to perform linear mapping on a key matrix and / or a value matrix corresponding to the historical time series data based on a preset linear mapping matrix, and the self-correlation attention submodule is configured to perform self-correlation mapping on one or more of a query matrix, the key matrix and the value matrix corresponding to the historical time series data. The device is further configured to input the historical time series data as input data into a prediction model, convert the historical time series data into a vector form by using an input embedding layer, obtain an input data vector and a global time feature vector of the historical time series data, determine a position encoding feature vector of the historical time series data by using a position encoding layer, obtain an input representation vector based on the input data vector, the global time feature vector and the position encoding feature vector, input the input representation vector into a linear mapping self-attention module of the prediction model, and obtain a self-correlation feature of the historical time series data.

10. An electronic device, comprising: comprising: one or more processors; a memory device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-8.

11. A computer readable medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the method according to any one of claims 1-8.

Images