Water level height prediction method and system based on space expansion convolution and multi-head multivariable attention

Abstract

The invention relates to a water level height prediction method and system based on space expansion convolution and multi-head multivariable attention, and belongs to the technical field of data processing for prediction purposes, and the method comprises the steps: giving water level height time sequence data as an input sequence, and carrying out the operation of the data through reversible instance normalization; adding the time position information as a variable into an input sequence; the sequence containing the time information passes through a lightweight embedding layer; respectively training and predicting the data set by adopting a prediction method of space expansion convolution and multi-head multivariable attention; transmitting the prediction sequence to a feedforward neural network; and removing time position variables in a prediction result, and obtaining a final subway flooding height prediction result through anti-instance normalization. According to the method, exploration of a complex dependency relationship between variables is emphasized, features are captured from two dimensions of time and space at the same time, the defect of water level height time sequence variable dimension feature deficiency is overcome, and the prediction precision of the water level height is effectively improved.

Description

Technical Field

[0001] The present invention relates to a water level height prediction method and system based on spatial dilated convolution and multi-head multi-variable attention, belonging to the technical field of data processing for prediction purposes. Background Art

[0002] Subways are usually built underground or on the ground and are easily affected by changes in rainwater, groundwater, or river water levels. After a subway flooding disaster occurs, it is necessary to obtain the change in the flooding height of the track tunnel or platform area in real time, and based on the height, warn of possible flooding risks to prevent track damage, equipment failures, or vehicle blockages caused by flooding, and ensure the normal operation of the subway.

[0003] The prediction of subway flooding height has experienced a transformation from traditional prediction methods to machine learning-based methods and then to deep learning-based methods. Traditional prediction methods for subway flooding height include trend analysis, regression analysis, exponential smoothing method, grey prediction model, etc. There are many factors affecting the flooding height, and it is difficult for traditional numerical simulation methods to accurately predict. Therefore, machine learning-based subway flooding height prediction methods have emerged. Machine learning has strong non-linear mapping capabilities and can effectively handle non-linear problems in subway flooding height prediction. Traditional machine learning methods include support vector machines, decision trees, random forests, etc. However, traditional machine learning methods still have some deficiencies in feature mining. Deep learning-based methods can fully describe the complex features of influencing factors, and their adaptive learning characteristics are conducive to fitting these complex features. However, some existing deep learning-based subway flooding height prediction methods still have the following problems:

[0004] (1) The subway flooding height is greatly affected by the region, and it is difficult for the prediction model to extract the features of the subway flooding height. Moreover, the instability in the subway flooding height will make the prediction results inaccurate.

[0005] (2) The model generalization ability is insufficient, and it is unable to extract the dependency relationship of subway flooding height time series data relatively completely, manifested as poor performance and large prediction errors.

[0006] (3) The modeling process is complex, with high requirements for the stationarity of the original time series, and the external factors affecting the load change are not fully considered.

[0007] (4) Previous prediction methods focused on extracting feature relationships from the time dimension of the sequence and lacked the capture of the dependency relationship between variables. Summary of the Invention

[0008] In view of the deficiencies of the prior art, in order to solve the problem of water level height prediction, the present invention provides a water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention to solve the problems raised in the above background, effectively reducing the error of subway flood height prediction, and can also be used in similar scenarios such as reservoir water level prediction.

[0009] The technical solution of the present invention is as follows:

[0010] A water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention includes the following steps:

[0011] Step 1, given the water level height time series data as the input sequence, the data set is input in segments. The data set is divided into M segments, trained and predicted together, and the data is processed using reversible instance normalization.

[0012] Step 2, add the time position information as a variable to the input sequence.

[0013] Step 3, the input sequence containing time information passes through a lightweight embedding layer and then is input into the encoder network model.

[0014] Step 4, use the prediction method of spatial dilated convolution and multi-head multi-variable attention to train and predict the M segments of sequences; each segment of sequence is trained and predicted as a whole through the entire encoder network model. The encoder network model includes the following modules:

[0015] 4-1) The input sequence enters the spatial dilated convolution layer, which includes three spatial dilated convolution blocks. The spatial dilated convolution block includes dilated convolution, layer normalization, GeLU activation function, and dropout layer.

[0016] 4-2) The input sequence enters the multi-head multi-variable attention layer, which mainly includes multi-head multi-variable self-attention, including: input, multi-head multi-variable self-attention, output. The multi-head multi-variable self-attention performs multi-head splitting, attention calculation, and multi-head merging operations.

[0017] 4-3) The input sequence enters the time multi-layer perceptron network, which includes a feed-forward neural layer, an activation function layer, and a feed-forward neural layer.

[0018] Step 5, transfer the prediction sequence obtained in Step 4 to the feed-forward neural network layer to embed the time dimension. It is the feed-forward neural network layer behind the encoder. This network layer maps the input data from d dimensions to the desired prediction layer H through linear transformation and non-linear activation.

[0019] Step 6, remove the time position variable in the prediction result, and obtain the final water level height prediction result through inverse instance normalization.

[0020] Preferably, step 1 includes:

[0021] The historical water level height time series data X ∈ R C×L , where C represents the channel, which is a variable; L represents the time length, and reversible instance normalization (RevIN) is used to mitigate the impact of distribution shift on the prediction performance:

[0022]

[0023] where γ is the learnable parameter for normalization, β is the learnable parameter for affine transformation, E[X] and Var[X] respectively represent the mean and variance of the input time series X, and ∈ is a constant. is the sequence after normalization.

[0024] Preferably, step 2 includes:

[0025] Add the time position information T ∈ R t×L as a variable to the input sequence to obtain X t ∈ R (C+t)×L , where t represents the number of elements in the time vector.

[0026]

[0027] Preferably, step 3 includes:

[0028] The lightweight embedding layer includes an input layer, a hidden layer, and an output layer. Through the lightweight embedding layer, X t ∈ R (C+t)×L is embedded into X emb ∈ R (C+t)×d , that is, this mapping transforms the time dimension from L to d, and W t ∈ R L×d is a learnable weight matrix, and b t is the bias vector;

[0029] X emb = X t W t + b t (3).

[0030] Preferably, step 4-1 includes:

[0031] After embedding the input sequence, the generated X emb first passes through spatial dilated convolution, and the output end of the embedding is connected to the input end of the dilated convolution.

[0032]

[0033] In the formula, K is the size of the convolution kernel, r is the dilation rate, and ω kis the weight of the convolution kernel, V i is the i-th variable among all variables; y i is the output after the convolution operation; the output end of the dilated convolution is connected to the input end of the weight normalization;

[0034] the output end of the weight normalization is connected to the input end of the GeLU activation function:

[0035]

[0036] where tanh(·) represents the hyperbolic tangent function and π represents the mathematical constant;

[0037] the output end of the GeLU activation function is connected to the output end of the dropout layer;

[0038] At least two spatial dilated convolution layers are adopted, and the input of the first layer and the output of the last layer have the same number of channels, and the residual connection is directly used.

[0039] Preferably, the step 4-2 includes:

[0040] the output end of the input layer is connected to the input end of the multi-head multi-variate self-attention:

[0041] The query of each vector is obtained through linear transformation key and value The vector is divided into h heads, and each head has its own weight matrix;

[0042]

[0043] where, is the weight matrix of the linear transformation in each attention head;

[0044] For each attention head, calculate the attention weight α i ,

[0045]

[0046] Calculate the weighted sum of the values, summarize the information from different variables at the same time, connect the outputs of each attention head, and apply a linear transformation to the connected output to restore it to the same dimension as the input;

[0047] head i = α i V i ,

[0048] MultiHead(X) = Concat(head1,head2...,head h )W O(15)

[0049] Among them, Concat(·) represents the concatenation operation, and W o is the output weight matrix.

[0050] Preferably, the step 4-3 includes:

[0051] The output end of the multi-head multi-variate attention layer is connected to the input end of the time multi-layer perceptron network;

[0052] The input of the first layer

[0053]

[0054] Among them, and are weight matrices for learning time pattern information, and are bias vectors, dff is the configurable hidden layer size, and σ is the GeLU activation function.

[0055] Preferably, the step 5 includes:

[0056] Transmit the high-dimensional time representation into the feed-forward neural network layer, and this network layer maps the input data from d dimensions to the desired prediction layer H through linear transformation and non-linear activation:

[0057]

[0058] Among them, among them is the basis for generating the prediction result after being adjusted by the feed-forward neural network layer.

[0059] Preferably, the step 6 includes:

[0060] Delete the added time variable, and map the variable dimension from (C + t) to C;

[0061] Return the prediction sequence to its original distribution through the inverse RevIN to obtain the final output:

[0062]

[0063] Among them, is the final prediction sequence.

[0064] A computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, it implements the steps in a water level height prediction method based on spatial dilated convolution and multi-head multi-variate attention as described above.

[0065] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the aforementioned water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention are implemented.

[0066] The beneficial effects of the present invention are as follows:

[0067] (1) In this method, the feature mapping dimension is changed from time to variable dimension, and then spatial dilated convolution is used to capture longer-term variable dependencies through a larger receptive field and extract missing variable dimension information.

[0068] (2) The present invention proposes a water level height prediction method that extracts features from both the time dimension and the variable dimension simultaneously, which can capture variable dependencies more effectively. To capture the dependencies between variables in the water level height time series data, local variable features and global variable features are extracted through a spatial dilated convolution layer and a multi-variate multi-head attention layer respectively, and time features are captured using the multi-head attention mechanism and a time multi-layer perceptron.

[0069] (3) In the prediction process of this method, the relationships between different variables are learned instead of treating them as independent features. This method uses the multi-head mechanism to learn the dependencies between variables in different periods, and accurately captures the global complex relationships between variables by calculating the correlations between each variable and all other variables, thereby better capturing the dependencies of the subway water flooding height time series and improving the prediction accuracy. Description of the Drawings

[0070] Figure 1 It is a schematic flow chart of the water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention of the present invention;

[0071] Figure 2 It is a schematic flow chart of the encoder network model of the water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention of the present invention;

[0072] Figure 3 It is a schematic flow chart of the spatial dilated convolution layer of the water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention of the present invention;

[0073] Figure 4 It is a schematic flow chart of the multi-head multi-variable attention layer of the water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention of the present invention;

[0074] Figure 5 It is a schematic flow chart of the time multi-layer perceptron network of the water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention of the present invention. Detailed Embodiments

[0075] The technical solutions in the embodiments of the present invention will be clearly introduced below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0076] Embodiment 1:

[0077] Please refer to Figures 1-5 , a water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention, comprising the following steps:

[0078] Step 1, given the water level height time series data as the input sequence, the data set is input in segments. The data set is divided into M segments, trained and predicted together, and the data is processed using reversible instance normalization.

[0079] The historical water level height time series data X ∈ R C×L , where C represents the channel, which is a variable; L represents the time length, and reversible instance normalization RevIN is used to reduce the impact of distribution shift on the prediction performance:

[0080]

[0081] where γ is the learnable parameter for normalization, β is the learnable parameter for affine transformation, E[X] and Var[X] respectively represent the mean and variance of the input time series X, ∈ is a constant, is the sequence after normalization.

[0082] Step 2, add the time position information as a variable to the input sequence.

[0083] Add the time position information T ∈ R t×L as a variable to the input sequence to obtain X t ∈ R (C+t)×L , where t represents the number of elements in the time vector,

[0084]

[0085] Step 3, the input sequence containing time information passes through a lightweight embedding layer and then is input into the encoder network model.

[0086] The lightweight embedding layer includes an input layer, a hidden layer, and an output layer. Through the lightweight embedding layer, X t ∈ R (C+t)×L is embedded into X emb ∈ R (C+t)×d , that is to say, this mapping transforms the time dimension from L to d, Wt ∈R L×d is a learnable weight matrix, b t is the bias vector;

[0087] X emb = X t W t + b t (3)

[0088] The process of the l-th (l ∈ {1,..., N}) encoder network

[0089]

[0090] where represents the output of the i-th sub-layer of the encoder, represents the output of the l-th sub-layer of the encoder; SDC in the formula represents the dilated convolutional layer, MMA represents the multi-head multi-variate attention, and TMP represents the temporal multi-layer perceptron.

[0091] Step 4, use the prediction method of spatial dilated convolution and multi-head multi-variable attention to train and predict M sequences; each sequence is trained and predicted as a whole through the entire encoder network model, and the encoder network model includes the following modules:

[0092] 4-1) The input sequence enters the spatial dilated convolutional layer, and the spatial dilated convolutional layer includes three spatial dilated convolutional blocks. The specific structure is as Figure 3 , and this spatial dilated convolutional block includes dilated convolution, layer normalization, GeLU activation function, and dropout layer.

[0093] After embedding the input sequence, the generated X emb First, pass through the spatial dilated convolution. The output end of the embedding is connected to the input end of the dilated convolution,

[0094]

[0095] In the formula, K is the size of the convolution kernel, r is the dilation rate, ω k is the weight of the convolution kernel, V i is the i-th variable among all variables; y i is the output after the convolution operation; the output end of the dilated convolution is connected to the input end of the weight normalization;

[0096] The output end of the weight normalization is connected to the input end of the GeLU activation function:

[0097]

[0098] where tanh(·) represents the hyperbolic tangent function, and π represents the mathematical constant;

[0099] The output end of the GeLU activation function is connected to the output end of the dropout layer;

[0100] Two layers of dilated spatial convolution layers are adopted. Figure 3 As shown in the structural schematic of one layer of dilated spatial convolution layer, the input of the first layer and the output of the last layer have the same number of channels, and residual connection is directly used:

[0101] G n (x) = Dropout(μ(g j (x) + x)), j = 1 (9)

[0102] G n (x) = Dropout(μ(g j (G j-1 (x)) + G j-1 (x))), j ≥ 2 (10)

[0103] Among them, g(·) represents parts such as dilated convolution and weight normalization, μ represents the GeLU activation function; j is 2, indicating that there are a total of 2 dilated spatial convolution layers; G n (x) is the output.

[0104] 4 - 2) The input sequence enters the multi - head multi - variable attention layer, and the structure of this layer is as Figure 4 , and this layer mainly includes multi - head multi - variable self - attention, including: input, multi - head multi - variable self - attention, output, and the multi - head multi - variable self - attention performs multi - head splitting, attention calculation, and multi - head merging operations.

[0105] The output end of the input layer is connected to the input end of the multi - head multi - variable self - attention:

[0106] Queries for each vector are obtained through linear transformation keys and values The vector is divided into h heads, and each head has its own weight matrix;

[0107]

[0108] Among them, is the weight matrix of the linear transformation in each attention head;

[0109] For each attention head, attention weights α i are calculated,

[0110]

[0111] Calculate the weighted sum of the computed values, while summarizing the information from different variables, concatenate the outputs of each attention head, and apply a linear transformation to the concatenated output to restore it to the same dimension as the input;

[0112] head i = α i V i ,

[0113] MultiHead(X) = Concat(head1, head2..., head h )W O (15)

[0114] where Concat(·) represents the concatenation operation, and W o is the weight matrix of the output.

[0115] 4 - 3) The input sequence enters a time multi-layer perceptron network, which includes a feed-forward neural layer, an activation function layer, and a feed-forward neural layer.

[0116] The output end of the multi-head multi-attention layer is connected to the input end of the time multi-layer perceptron network;

[0117] The input of the first layer

[0118]

[0119] where and are weight matrices for learning time pattern information, and are bias vectors, dff is the configurable hidden layer size, and σ is the GeLU activation function.

[0120] Step 5, Pass the predicted sequence obtained in Step 4 to the feed-forward neural network layer for time dimension embedding, which is the feed-forward neural network layer behind the encoder. This network layer maps the input data from d dimensions to the desired prediction layer H through linear transformation and non-linear activation.

[0121] Pass the high-dimensional time representation into the feed-forward neural network layer, which maps the input data from d dimensions to the desired prediction layer H through linear transformation and non-linear activation:

[0122]

[0123] where is the basis for generating the prediction result after adjustment by the feed-forward neural network layer.

[0124] Step 6: Remove the time position variable from the prediction result, and obtain the final predicted water level height after inverse instance normalization.

[0125] Delete the added time variable, and map the variable dimension from (C + t) to C;

[0126] Return the prediction sequence to its original distribution through inverse RevIN to obtain the final output:

[0127]

[0128] where is the final prediction sequence.

[0129] The size of the embedded hidden layer is set to 512, and the number of encoders is set to 2. The spatial dilated convolution module consists of two layers. The dilation rates of the first layer of SDC are 1, 2, and 4, and the dilation rates of the second layer of spatial dilated convolution are 2, 4, and 8 respectively.

[0130] In the multi-head multi-variable attention, the number of multi-heads is set to 8.

[0131] The hidden unit df in the time multi-layer perceptron is 2048, and the drop-out rate is 0.05.

[0132] The mean square error (MSE) is used as the loss function in the training process, and the Adam optimizer is adopted.

[0133] The training stops within 10 epochs.

[0134] Use historical data for training. After the training is completed, the data to be predicted is returned to Step 1 for prediction.

[0135] To better experience the prediction results of the present invention, some indicators for evaluating regression problems are used in this embodiment to display the results, namely, mean square error (Mean Square Error), root mean square error (Root Mean Square Error), mean absolute error (Mean Absolute Error), and mean absolute percentage error (Mean Absolute Percentage Error).

[0136]

[0137] where y i is the real data at the i-th moment, is the predicted value of y of the model i and N is the data length.

[0138] The prediction method of the present invention was used to test and predict the subway waterlogging height dataset, and a relatively high accuracy was obtained. The average improvement rate was 9%, and the performance improvement was 35%.

[0139] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0140] The above content only illustrates the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution according to the technical idea proposed by the present invention fall within the protection scope of the claims of the present invention.

Claims

1. A water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention, characterized in that It includes the following steps: Step 1: Given the water level height time series data as the input sequence, the data set is input in segments. The data set is divided into M segments, trained and predicted together, and the data is processed using reversible instance normalization; Step 2: Add the time position information as a variable to the input sequence; Step 3: The input sequence containing time information passes through a lightweight embedding layer and then is input into the encoder network model; Step 4: Adopt a prediction method of spatial dilated convolution and multi-head multi-variable attention to train and predict the M segment sequences; each segment sequence is trained and predicted as a whole through the entire encoder network model. The encoder network model includes the following modules: 4-1) The input sequence enters the spatial dilated convolution layer. The spatial dilated convolution layer includes three spatial dilated convolution blocks, and each spatial dilated convolution block includes dilated convolution, layer normalization, GeLU activation function, and dropout layer; 4-2) The input sequence enters the multi-head multi-variable attention layer, including: input, multi-head multi-variable self-attention, output. The multi-head multi-variable self-attention performs multi-head splitting, attention calculation, and multi-head merging operations; 4-3) The input sequence enters the time multi-layer perceptron network, which includes a feed-forward neural layer, an activation function layer, and a feed-forward neural layer; Step 5: Pass the prediction sequence obtained in Step 4 to the feed-forward neural network layer to embed the time dimension. This network layer maps the input data from d dimensions to the desired prediction layer H through linear transformation and non-linear activation; Step 6: Remove the time position variable in the prediction result, and obtain the final prediction result of the water level height after inverse instance normalization.

2. The water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention according to claim 1, characterized in that The said Step 1 includes: The time series data of water level height \(X\in R\) C×L , where \(C\) represents the channel, which is a variable; \(L\) represents the time length, and the Reversible Instance Normalization (RevIN) is used to mitigate the impact of distribution shift on the prediction performance: where γ is a normalized learnable parameter, β is a learnable parameter for affine transformation, E[X] and Var[X] represent the mean and variance of the input time series X respectively, and ∈ is a constant, is the normalized sequence.

3. The water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention according to claim 1, characterized in that The said Step 2 includes: Add the time position information \(T\in\mathbb{R}\) t×L as a variable to the input sequence to obtain \(X\) t \(\in\mathbb{R}\) (C+t)×L , where \(t\) represents the number of elements of the time vector 4. The water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention according to claim 1, wherein The said Step 3 includes: The lightweight embedding layer includes an input layer, a hidden layer, and an output layer. Through the lightweight embedding layer, X t ∈R (C+t)×L is embedded into X emb ∈R (C+t)×d . This mapping transforms the time dimension from L to d. W t ∈R L×d is a learnable weight matrix, and b t is the bias vector; X emb = X t W t + b t (3).

5. The water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention according to claim 1, characterized in that The said Step 4-1 includes: After embedding the input sequence, the generated X emb First, through dilated spatial convolution, the output end of the embedding is connected to the input end of the dilated convolution. where K is the size of the convolution kernel, r is the dilation rate, ω k is the weight of the convolution kernel, V i is the i-th variable among all variables; y i is the output after the convolution operation; the output end of the dilated convolution is connected to the input end of the weight normalization; The output end of weight normalization is connected to the input end of the GeLU activation function: where tanh(·) represents the hyperbolic tangent function, and π represents the mathematical constant; The output end of the GeLU activation function is connected to the output end of the dropout layer; Adopt at least two layers of spatial dilated convolution layers. The input of the first layer and the output of the last layer have the same number of channels, and residual connection is directly used.

6. The water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention according to claim 1, wherein The said Step 4-2 includes: The output end of the input layer is connected to the input end of the multi-head multi-variable self-attention: Obtain the query of each vector through linear transformation Key and value The vector is divided into h heads, and each head has its own weight matrix; Among them, is the weight matrix of the linear transformation in each attention head; For each attention head, compute the attention weights α i , Calculate the weighted sum of the values, and at the same time summarize the information from different variables. Connect the outputs of each attention head, and apply a linear transformation to the connected output to restore it to the same dimension as the input; head i = α i V i , MultiHead(X)=Concat(head1,head2...,head h )W O (15) Among them, Concat(·) represents the concatenation operation, and W o is the output weight matrix.

7. The water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention according to claim 1, characterized in that The said Step 4-3 includes: The output end of the multi-head multi-variable attention layer is connected to the input end of the time multi-layer perceptron network; Input of the first layer Among them, and are weight matrices for learning time pattern information, and are bias vectors, dff is the configurable hidden layer size, and σ is the GeLU activation function.

8. The water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention according to claim 1, wherein The said Step 5 includes: Pass the high-dimensional time representation into the feed-forward neural network layer. This network layer maps the input data from d dimensions to the desired prediction layer H through linear transformation and non-linear activation: Among them, among them is the basis for generating the prediction result after being adjusted by the feedforward neural network layer.

9. The water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention according to claim 1, characterized in that The said Step 6 includes: Delete the added time variable, and map the variable dimension from (C + t) to C; Pass the prediction sequence back to its original distribution through the inverse RevIN to obtain the final output: Among them, is the final predicted sequence.

10. A computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, it implements the steps in the water level height prediction method based on spatial dilated convolution and multi-head multi-variable attention as described in any one of claims 1-9.

Images