Water level height prediction method and system based on spatial dilated convolution and multi-head multivariate attention

Abstract

The present application relates to a water level height prediction method and system based on spatial expansion convolution and multi-head multivariate attention, belonging to the data processing technical field for prediction purposes, comprising the steps: given water level height time series data as input sequence, using reversible instance normalization to operate on the data; time position information is added as a variable to the input sequence; the sequence containing time information passes through a lightweight embedding layer; the prediction method of spatial expansion convolution and multi-head multivariate attention is used to train and predict the data set respectively; the prediction sequence is transmitted to the feedforward neural network; the time position variable in the prediction result is removed, and the final subway water flooding height prediction result is obtained through inverse instance normalization. The present application emphasizes the exploration of complex dependency relationship between variables, and captures features from two dimensions of time and space, making up for the lack of water level height time series variable dimension feature, effectively improving the prediction accuracy of water level height.

Description

TECHNICAL FIELD

[0001] The present application relates to a water level height prediction method and system based on spatial dilated convolution and multi-head multivariate attention, belonging to the data processing technical field for prediction purposes. BACKGROUND

[0002] Subways are usually built underground or on the ground, and are easily affected by changes in rainwater, groundwater, or river water levels. After a subway water flooding disaster occurs, it is necessary to obtain the water flooding height changes of the track tunnel or platform area in real time, based on the height warning of possible water flooding risks, to prevent track damage, equipment failure, or vehicle obstruction caused by water flooding, and to ensure the normal operation of the subway.

[0003] Subway water flooding height prediction has undergone a transformation from traditional prediction methods to machine learning-based methods, and then to deep learning-based methods. Traditional prediction methods for subway water flooding height prediction include trend analysis, regression analysis, exponential smoothing, and gray prediction models. There are many factors that affect water flooding height, and it is difficult to accurately predict using traditional numerical simulation methods. Therefore, machine learning-based subway water flooding height prediction methods have emerged. Machine learning has strong non-linear mapping capabilities and can effectively handle non-linear problems in subway water flooding height prediction. Traditional machine learning methods include support vector machines, decision trees, and random forests, but traditional machine learning methods still have some shortcomings in feature mining. Deep learning-based methods can fully describe the complex characteristics of influencing factors, and the adaptive learning characteristics are beneficial for fitting these complex characteristics, but some existing deep learning-based subway water flooding height prediction methods still have the following problems:

[0004] (1) Subway water flooding height is greatly affected by the region, making it difficult to extract subway water flooding height features for the prediction model, and the instability of subway water flooding height can result in inaccurate prediction results.

[0005] (2) The model has insufficient generalization ability and cannot fully extract the dependency relationships of subway water flooding height time series data, resulting in poor performance and large prediction errors.

[0006] (3) The modeling process is complex, requires high stationarity of the original time series, and does not fully consider external factors that affect load changes.

[0007] (4) Previous prediction methods focused on extracting feature relationships from the time dimension of the sequence, and lacked the capture of dependency relationships between variables. SUMMARY

[0008] In view of the deficiencies of the prior art, in order to solve the water level height prediction problem, the application provides a water level height prediction method based on spatial expansion convolution and multi-head multivariate attention to solve the problems in the above background, effectively reduces the error of subway water flooding height prediction, and can also be used for reservoir water level prediction and similar scenes.

[0009] The technical scheme of the application is as follows:

[0010] The water level height prediction method based on spatial expansion convolution and multi-head multivariate attention comprises the following steps:

[0011] Step 1, given the water level height time series data as the input sequence, the data set adopts segmented input, and the data set is divided into M segments, which are trained and predicted together, and reversible instance normalization is used for data processing operation;

[0012] Step 2, the time position information is added as a variable to the input sequence;

[0013] Step 3, the input sequence containing the time information is input into an encoder network model through a lightweight embedding layer;

[0014] Step 4, the spatial expansion convolution and multi-head multivariate attention prediction method is used to train and predict the M sequences; each sequence is trained and predicted as a whole through the entire encoder network model, and the encoder network model comprises the following modules:

[0015] 4-1) the input sequence enters a spatial expansion convolution layer, the spatial expansion convolution layer comprises three spatial expansion convolution blocks, and the spatial expansion convolution block comprises an expansion convolution, a layer normalization, a GeLU activation function and a dropout layer;

[0016] 4-2) the input sequence enters a multi-head multivariate attention layer, which mainly comprises multi-head multivariate self-attention, including: input, multi-head multivariate self-attention, output, multi-head multivariate self-attention, multi-head segmentation, attention calculation and multi-head merging operation;

[0017] 4-3) the input sequence enters a time multilayer perception network, which comprises a feedforward neural layer, an activation function layer and a feedforward neural layer;

[0018] Step 5, the prediction sequence obtained in step 4 is transmitted to a feedforward neural network layer, and the time dimension is embedded, which is a feedforward neural network layer behind the encoder, and the network layer maps the input data from d dimensions to the expected prediction layer H through linear transformation and nonlinear activation;

[0019] Step 6, remove the time position variable in the prediction result, and obtain the final water level height prediction result through inverse instance normalization.

[0020] Preferably, the step 1 comprises:

[0021] Historical water level height time series data X ∈ R C×L where C denotes the channel, L denotes the length of time, and RevIN is used to mitigate the impact of distribution shift on the prediction performance:

[0022]

[0023] where γ is a normalized learnable parameter, β is a learnable parameter for affine transformation, E[X] and Var[X] denote the mean and variance of the input time series X respectively, ∈ is a constant, is the normalized sequence.

[0024] Preferably, the step 2 comprises:

[0025] The time position information T ∈ R t×L is added to the input sequence as a variable to obtain X t ∈ R (C+t)×L where t denotes the number of time vector elements,

[0026]

[0027] Preferably, the step 3 comprises:

[0028] The lightweight embedding layer includes an input layer, a hidden layer and an output layer, and X t ∈ R (C+t)×L is embedded into X emb ∈ R (C+t)×d That is, this mapping transforms the time dimension from L to d, W t ∈ R L×d is a learnable weight matrix, and b t is a bias vector;

[0029] X emb = X t W t +b t (3).

[0030] Preferably, the step 4-1 comprises:

[0031] After embedding the input sequence, the generated X emb is first connected to the input end of the dilated convolution through spatial dilated convolution,

[0032]

[0033] where K is the size of the convolution kernel, r is the dilation rate, and ω kV is the weight of the convolution kernel i is the i-th variable among all variables; y i is the output after the convolution operation; the output end of the dilated convolution is connected to the input end of the weight normalization;

[0034] The output end of the weight normalization is connected to the input end of the GeLU activation function:

[0035]

[0036] where tanh(·) represents the hyperbolic tangent function, and π represents a mathematical constant;

[0037] The output end of the GeLU activation function is connected to the output end of the dropout layer.

[0038] At least two layers of spatial dilated convolution layers are used, the input of the first layer and the output of the last layer have the same number of channels, and a residual connection is directly used.

[0039] Preferably, the step 4-2 comprises:

[0040] The output end of the input layer is connected to the input end of the multi-head multi-element self-attention:

[0041] The query of each vector is obtained by linear transformation Key and value The vector is divided into h heads, each head has its own weight matrix;

[0042]

[0043] where, is the weight matrix of the linear transformation in each attention head;

[0044] For each attention head, the attention weight α i is calculated

[0045]

[0046] The weighted sum of the values is calculated, the information from different variables is summarized, the outputs of each attention head are connected, and a linear transformation is applied to the connected output to restore it to the same dimension as the input.

[0047] head i = α i V i ,

[0048] MultiHead(X)=Concat(head1,head2...,head h )W O(15)

[0049] where Concat(·) denotes the concatenation operation, W o is the weight matrix of the output.

[0050] Preferably, the step 4-3 comprises:

[0051] The output of the multi-head multi-variable attention layer is connected to the input of the temporal multi-layer perception network.

[0052] The input of the first layer

[0053]

[0054] where, and are the weight matrices for learning temporal pattern information, and are the bias vectors, dff is the configurable hidden layer size, and σ is the GeLU activation function.

[0055] Preferably, the step 5 comprises:

[0056] The high-dimensional temporal representation is passed into a feed-forward neural network layer, which maps the input data from d dimensions to the desired prediction layer H through linear transformations and non-linear activations:

[0057]

[0058] where, where is the basis for generating a prediction result after adjustment by the feed-forward neural network layer.

[0059] Preferably, the step 6 comprises:

[0060] The added temporal variable is deleted, and the variable dimension is mapped from (C+t) to C;

[0061] The prediction sequence is returned to its original distribution through the inverse RevIN to obtain the final output:

[0062]

[0063] where, is the final prediction sequence.

[0064] A computer-readable storage medium having a program stored thereon, the program being executed by a processor to implement the steps of a water level height prediction method based on spatial dilation convolution and multi-head multi-variable attention as described above.

[0065] An electronic device includes a memory, a processor, and a program stored on the memory and executable on the processor, and the processor implements the steps of the water level height prediction method based on spatial dilated convolution and multi-head multivariate attention as described above when executing the program.

[0066] The present application has the advantages of:

[0067] (1) The method changes the feature mapping dimension from time to variable dimension, then uses spatial dilated convolution to capture longer-term variable dependencies through a larger receptive field, and extracts missing variable dimension information.

[0068] (2) The present application proposes a water level height prediction method that extracts features from both time and variable dimensions simultaneously, which can more effectively capture variable dependencies. In order to capture the dependencies between variables in water level height time series data, local variable features and global variable features are extracted through spatial dilated convolution layers and multivariate multi-head attention layers, respectively, and time features are captured using multi-head attention mechanisms and time multi-layer perceptrons.

[0069] (3) The method learns the relationships between different variables during the prediction process, rather than treating them as independent features. The method uses a multi-head mechanism to learn the dependencies between variables at different times, accurately captures the global complex relationships between variables by calculating the correlation between each variable and all other variables, thereby better capturing the dependencies of subway water flooding height time series and improving prediction accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 The flowchart of the water level height prediction method based on spatial dilated convolution and multi-head multivariate attention of the present application;

[0071] Figure 2 The flowchart of the encoder network model of the water level height prediction method based on spatial dilated convolution and multi-head multivariate attention of the present application;

[0072] Figure 3 The flowchart of the spatial dilated convolution layer of the water level height prediction method based on spatial dilated convolution and multi-head multivariate attention of the present application;

[0073] Figure 4 The flowchart of the multi-head multivariate attention layer of the water level height prediction method based on spatial dilated convolution and multi-head multivariate attention of the present application;

[0074] Figure 5 The flowchart of the time multi-layer perceptron network of the water level height prediction method based on spatial dilated convolution and multi-head multivariate attention of the present application. DETAILED DESCRIPTION

[0075] The technical solutions in the embodiments of the present application will be clearly introduced below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, other embodiments obtained by a person of ordinary skill in the art without creative labor are within the protection scope of the present application.

[0076] Embodiment 1

[0077] Please refer to Figures 1-5 , the water level height prediction method based on spatial expansion convolution and multi-head multivariate attention, including the following steps:

[0078] Step 1, given the water level height time series data as the input sequence, the data set adopts segmented input, the data set is divided into M segments, which are trained and predicted together, and reversible instance normalization is used for data processing operation.

[0079] The historical water level height time series data X ∈ R C×L , where C represents the channel, L represents the time length, and reversible instance normalization RevIN is used to reduce the influence of distribution deviation on prediction performance:

[0080]

[0081] Where γ is the normalized learnable parameter, β is the learnable parameter of affine transformation, E[X] and Var[X] represent the mean and variance of the input time series X respectively, ∈ is a constant, is the normalized sequence.

[0082] Step 2, the time position information is added as a variable to the input sequence.

[0083] The time position information T ∈ R t×L is added as a variable to the input sequence, and X t ∈ R (C+t)×L , where t represents the number of time vector elements,

[0084]

[0085] Step 3, the input sequence containing time information is input into the encoder network model through a lightweight embedding layer.

[0086] The lightweight embedding layer includes an input layer, a hidden layer and an output layer, X t ∈ R (C+t)×L is embedded into X emb ∈ R (C+t)×d , that is, this mapping changes the time dimension from L to d, Wt ∈R L×d is a learnable weight matrix, b t is a bias vector;

[0087] X emb = X t W t +b t (3)

[0088] Process of the lth (l ∈ {1,..., N}) encoder network

[0089]

[0090] wherein, represents the output of the ith sub-layer of the encoder, represents the output of the lth sub-layer of the encoder; SDC in the formula represents a dilated convolution layer, MMA represents a multi-head multi-variable attention, and TMP represents a temporal multi-layer perceptron.

[0091] Step 4, training and prediction of the M sequences by using the prediction method of spatial dilated convolution and multi-head multi-variable attention; each sequence is trained and predicted as a whole through the entire encoder network model, and the encoder network model includes the following modules:

[0092] 4-1) The input sequence enters a spatial dilated convolution layer, and the spatial dilated convolution layer includes three spatial dilated convolution blocks, and the specific structure is as shown in Figure 3 The spatial dilated convolution block includes a dilated convolution, a layer normalization, a GeLU activation function and a dropout layer.

[0093] After embedding the input sequence, X emb is generated, and the output end of the spatial dilated convolution is connected to the input end of the dilated convolution,

[0094]

[0095] In the formula, K is the size of the convolution kernel, r is the dilated rate, ω k is the weight of the convolution kernel, V i is the ith variable in all variables; y i is the output after the convolution operation; the output end of the dilated convolution is connected to the input end of the weight normalization;

[0096] The output end of the weight normalization is connected to the input end of the GeLU activation function:

[0097]

[0098] Wherein tanh(·) represents a hyperbolic tangent function, and π represents a mathematical constant;

[0099] The output end of the GeLU activation function is connected to the output end of the dropout layer;

[0100] Two layers of spatial dilated convolution layers are adopted, Figure 3 For the structure diagram of a layer of spatial dilated convolution layer, the input of the first layer and the output of the last layer have the same number of channels, and a residual connection is directly used:

[0101] G n (x) = Dropout(μ(g j (x) + x)), j = 1 (9)

[0102] G n (x) = Dropout(μ(g j (G j-1 (x)) + G j-1 (x))), j ≥ 2 (10)

[0103] Wherein, g(·) represents dilated convolution and weight normalization, etc., μ represents GeLU activation function; j is 2, indicating that there are 2 spatial dilated convolution layers in total; G n (x) is the output.

[0104] 4-2) The input sequence enters the multi-head multi-variable attention layer, and the structure of the layer is as follows Figure 4 The layer mainly includes multi-head multi-variable self-attention, including: input, multi-head multi-variable self-attention, output, multi-head multi-variable self-attention, multi-head segmentation, attention calculation, and multi-head merging operation.

[0105] The output end of the input layer is connected to the input end of the multi-head multi-variable self-attention:

[0106] The query key and value Vectors are divided into h heads, and each head has its own weight matrix;

[0107]

[0108] Wherein, is the weight matrix of linear transformation in each attention head;

[0109] For each attention head, the attention weight α i is calculated.

[0110]

[0111] computing a weighted sum of the values, summarizing information from different variables, concatenating the outputs of each attention head, and applying a linear transformation to the concatenated output to restore it to the same dimensionality as the input;

[0112] head i = a i V i ,

[0113] MultiHead(X) = Concat(head1, head2..., head h )W O (15)

[0114] where Concat(·) denotes the concatenation operation and W o is the weight matrix for the output.

[0115] 4-3) The input sequence enters a temporal multi-layer perception network that includes a feedforward neural layer, an activation function layer, and a feedforward neural layer.

[0116] The output of the multi-head multi-element attention layer is connected to the input of the temporal multi-layer perception network;

[0117] Input of the first layer

[0118]

[0119] where, and are weight matrices that learn temporal pattern information, and are bias vectors, dff is a configurable hidden layer size, and σ is a GeLU activation function.

[0120] Step 5: The predicted sequence obtained in step 4 is passed to a feedforward neural network layer for embedding in the time dimension, which is a feedforward neural network layer behind the encoder. This network layer maps input data from d dimensions to the desired prediction layer H through linear transformation and nonlinear activation.

[0121] The high-dimensional temporal representation is passed to a feedforward neural network layer that maps input data from d dimensions to the desired prediction layer H through linear transformation and nonlinear activation:

[0122]

[0123] where, is the basis for generating a prediction result after adjustment by the feedforward neural network layer.

[0124] Step 6: Remove the time location variable from the prediction results and obtain the final water level prediction result after inverse instance normalization.

[0125] Remove the added time variable and map the variable dimension from (C+t) to C;

[0126] The predicted sequence is then reversed using RevIN to return it to its original distribution, yielding the final output:

[0127]

[0128] in, This is the final predicted sequence.

[0129] The size of the embedded hidden layer is set to 512, and the number of encoders is set to 2. The spatially dilated convolution module consists of two layers, where the first layer SDC has a dilation rate of 1, 2, and 4, and the second layer spatially dilated convolution has a dilation rate of 2, 4, and 8.

[0130] In the multi-head multivariate attention, the number of long positions is set to 8.

[0131] The hidden unit depth function (df) in the temporal multilayer perceptron is 2048, and the drop-out rate is 0.05.

[0132] The training process uses mean squared error (MSE) as the loss function and employs the Adam optimizer.

[0133] Training was stopped after 10 cycles.

[0134] The system is trained using historical data. After training, the data to be predicted is returned to step 1 for prediction.

[0135] To better illustrate the prediction results of this invention, this embodiment uses several metrics for evaluating regression problems to display the results, namely Mean Square Error, Root Mean Square Error, Mean Absolute Error, and Mean Absolute Percentage Error.

[0136]

[0137] Where y i For the actual data at time i, For the model's y i Predicted value, where N is the data length.

[0138] The prediction method of the application is used to test and predict the subway water flooding height data set, and a higher accuracy is obtained. The average improvement rate is 9%, and the performance is improved by 35%.

[0139] Although the preferred embodiments of the application have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the preferred embodiments and all changes and modifications falling within the scope of the application.

[0140] The above is only to illustrate the technical idea of the application, and cannot limit the protection scope of the application. Any modification made on the basis of the technical scheme according to the technical idea of the application falls within the protection scope of the claims of the application.

Claims

1. A water level prediction method based on spatially dilated convolution and multi-head multivariate attention, characterized in that, The steps include the following: Step 1: Given water level height time series data as input sequence, the dataset is segmented into M segments for training and prediction, and reversible instance normalization is used to process the data. Step 2: Add the time and location information as variables to the input sequence; Step 3: The input sequence containing time information is passed through a lightweight embedding layer and then fed into the encoder network model; Step 4: The prediction method of spatially dilated convolution and multi-head multivariate attention is used to train and predict the M segments of the sequence; each segment of the sequence is trained and predicted as a whole through the entire encoder network model, which includes the following modules: 4-1) The input sequence enters the spatially dilated convolutional layer, which includes three spatially dilated convolutional blocks, each of which includes dilated convolution, layer normalization, GeLU activation function, and dropout layer; 4-2) The input sequence enters the multi-head multi-dimensional attention layer, including: input, multi-head multi-dimensional self-attention, output, and multi-head multi-dimensional self-attention performs multi-head segmentation, attention calculation, and multi-head merging operations; 4-3) The input sequence enters the time-of-use multilayer perceptron, which includes a feedforward neural layer, an activation function layer, and a feedforward neural layer; Step 5: Pass the predicted sequence obtained in step 4 to the feedforward neural network layer to embed the time dimension. This network layer maps the input data from d dimensions to the desired prediction layer H through linear transformation and nonlinear activation. Step 6: Remove the time location variable from the prediction results and obtain the final water level prediction result after inverse instance normalization.

2. The water level prediction method based on spatially dilated convolution and multi-head multivariate attention as described in claim 1, characterized in that, Step 1 includes: Water level height time series data X∈R C×L Where C represents the channel and is a variable; L represents the time length. Reversible instance normalization RevIN is used to mitigate the impact of distribution offset on prediction performance. Where γ is a normalized learnable parameter, β is a learnable parameter of the affine transformation, E[X] and Var[X] represent the mean and variance of the input time series X, respectively, and ∈ is a constant. This is the normalized sequence.

3. The water level prediction method based on spatially dilated convolution and multi-head multivariate attention as described in claim 1, characterized in that, Step 2 includes: Time location information T∈R t×L Added as a variable to the input sequence, resulting in X. t ∈R (C+t)×L , where t represents the number of elements in the time vector.

4. The water level prediction method based on spatially dilated convolution and multi-head multivariate attention as described in claim 1, characterized in that, Step 3 includes: The lightweight embedding layer consists of an input layer, a hidden layer, and an output layer. X is embedded through this lightweight embedding layer. t ∈R (C+t)×L Embedded into X emb ∈R (C+t)×d This mapping transforms the time dimension from L to d, W t ∈R L×d It is a learnable weight matrix, b t It is the bias vector; X emb =X t W t +b t (3)。 5. The water level prediction method based on spatially dilated convolution and multi-head multivariate attention as described in claim 1, characterized in that, Step 4-1 includes: After embedding the input sequence, the generated X emb First, spatial dilation convolution is performed, and the embedded output is connected to the input of the dilation convolution. In the formula, K is the size of the convolution kernel, r is the dilation rate, and ω k V represents the weights of the convolution kernel. i Let y be the i-th variable among all variables; i This is the output after the convolution operation; the output of the dilated convolution is connected to the input of the weight normalization. The output of the weighted normalized function is connected to the input of the GeLU activation function: Where tanh(·) represents the hyperbolic tangent function, and π represents a mathematical constant; The output of the GeLU activation function is connected to the output of the dropout layer; At least two spatially dilated convolutional layers are used, with the input of the first layer and the output of the last layer having the same number of channels, and residual connections are used directly.

6. The water level prediction method based on spatially dilated convolution and multi-head multivariate attention as described in claim 1, characterized in that, Step 4-2 includes: The output of the input layer is connected to the input of the multi-head, multi-dimensional self-attention layer: The query for each vector is obtained through linear transformation. key Sum The vector is divided into h heads, each head having its own weight matrix; in, It is the weight matrix of the linear transformation in each attention head; For each attention head, calculate the attention weight α. i , The weighted sum of the values ​​is calculated, while summarizing the information from different variables. The outputs of each attention head are concatenated, and a linear transformation is applied to the concatenated outputs to restore them to the same dimension as the input. head i =α i V i , MultiHead(X)=Concat(head1,head2...,head h )W O (15) Where Concat(·) represents a join operation, W o It is the output weight matrix.

7. The water level prediction method based on spatially dilated convolution and multi-head multivariate attention as described in claim 1, characterized in that, Step 4-3 includes: The output of the multi-head multi-dimensional attention layer is connected to the input of the temporal multilayer perceptual network; Input of Level 1 in, and The weight matrix for learning time pattern information. and is the bias vector, dff is the configurable hidden layer size, and σ is the GeLU activation function.

8. The water level prediction method based on spatially dilated convolution and multi-head multivariate attention as described in claim 1, characterized in that, Step 5 includes: The high-dimensional temporal representation is passed to a feedforward neural network layer, which maps the input data from d dimensions to the desired prediction layer H through linear transformations and nonlinear activations. Among them, It is the basis for generating prediction results after adjustments are made through feedforward neural network layers.

9. The water level prediction method based on spatially dilated convolution and multi-head multivariate attention as described in claim 1, characterized in that, Step 6 includes: Remove the added time variable and map the variable dimension from (C+t) to C; The predicted sequence is then reversed using RevIN to return it to its original distribution, yielding the final output: in, This is the final predicted sequence.

10. A computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of the water level height prediction method based on spatially dilated convolution and multi-head multivariate attention as described in any one of claims 1-9.

Images