Hydropower station water turbine top cover long sequence water level prediction system and method based on self-attention mechanism

Abstract

The application discloses a kind of long sequence water level prediction system and method of water turbine top cover of hydropower station based on self-attention mechanism, based on self-attention mechanism, embedding representation is processed to obtain attention feature matrix;Then using pooling operation is handled to attention feature matrix, generates encoding hidden representation;Further based on self-attention mechanism, encoding hidden representation is processed to obtain decoding hidden representation;Finally, based on neural network ODE, decoding hidden representation is processed to obtain prediction hidden state sequence, further by full connection processing obtains final prediction result, i.e. long sequence water level of water turbine top cover of hydropower station.The application can better capture the association between multiple observation values and the dependence on time for a long time, so it has higher water level prediction accuracy.

Description

Technical Field

[0001] This invention belongs to the field of water conservancy and hydropower technology, specifically relating to the monitoring of hydropower stations, and more specifically to a long-sequence water level prediction technology for the turbine roof of a hydropower station based on deep learning. This invention mainly utilizes deep learning methods to construct a prediction system that combines three techniques: self-attention, pooling, and Neural Ordinary Differential Equations (NODE). With an encoder and decoder as the main components, it incorporates self-attention and pooling modules, and adds Neural ODE during the prediction phase. In the task of predicting the water level of the turbine roof, this prediction method can learn the long-term dependency characteristics of multidimensional variables, enhancing the accuracy of the prediction results and achieving the goal of dynamically predicting the water level of the turbine roof at different time periods based on multidimensional variables. Background Technology

[0002] Hydropower stations generate electricity using the potential energy of water stored in reservoirs, making them a vital renewable energy source. Among the many components of a hydropower station, the turbine is one of the key pieces of equipment. Figure 1 As shown, the water level in the turbine's top cover changes continuously over time due to factors such as leakage in the turbine's main shaft sealing device and pump drainage. The turbine's top cover water level is a crucial factor affecting the turbine's operational safety. Excessively high water levels can flood the water guide bearings and oil tanks, causing oil deterioration and seriously threatening equipment safety. Timely monitoring and early warning of the top cover water level allows for analysis of the operating efficiency of the top cover pumps, ensuring the water level remains within a safe and controllable range, and improving the safety of the hydropower station's operation. Therefore, predicting the turbine's top cover water level is an important research topic.

[0003] Predicting water level changes in the turbine roof of a hydropower station is a typical long-sequence time series forecasting problem. Existing methods for time series forecasting can be broadly categorized into two types: classical models and deep learning methods. Classical time series models, based on statistical learning methods, offer advantages such as strong interpretability and comprehensive theoretical support, making them the primary approach for time series forecasting. However, classical time series models require substantial empirical data, resulting in high computational costs and low efficiency; furthermore, they cannot capture long-term dependencies, making them difficult to apply effectively to long-sequence water level forecasting for turbine roofs. In recent years, deep learning technology has developed rapidly. This technology primarily utilizes recurrent neural networks and their various variants, employing a sequence-to-sequence forecasting paradigm, achieving breakthroughs in forecasting performance. However, this method still cannot achieve accurate long-sequence forecasting, with problems manifested in poor parallel processing capabilities and weak ability to capture long-term dependencies.

[0004] In summary, developing a method to predict the water level of the turbine roof of a hydropower station over a long sequence is one of the key issues that urgently needs to be addressed in the current monitoring of hydropower station turbines. Summary of the Invention

[0005] In view of the current technical situation where it is difficult to effectively predict the long-term water level of the turbine roof, the present invention aims to provide a long-term water level prediction system for the turbine roof of a hydropower station based on a self-attention mechanism. This system can capture long-term dependency information of a long-term series based on the historical water level of the turbine roof and related characteristics (turbine unit flow rate, roof vibration, etc.) and accurately predict the water level of the turbine roof over a long period of time in the future.

[0006] Another objective of this invention is to provide a method for predicting the long sequence of water levels on the turbine roof of a hydropower station based on a self-attention mechanism.

[0007] The invention's core concept is to construct a prediction system based on a self-attention mechanism, enabling it to capture long-term dependency information of long-term sequences. Pooling operations are employed in the system implementation, significantly reducing the computational complexity of long-sequence prediction. Simultaneously, leveraging the advantages of deep neural networks, the system minimizes long-term prediction errors caused by iterative predictions, maintaining good accuracy across different time periods. Furthermore, the self-attention mechanism eliminates the sequential structure of the system, resulting in better parallelism. In addition, the integration of neural ODEs further enhances the dynamic nature of the prediction process.

[0008] Based on the above-mentioned inventive concept, the present invention provides a long-sequence water level prediction system for the turbine roof of a hydropower station based on a self-attention mechanism, which includes:

[0009] The data preprocessing module is used to preprocess the water level and related data of the turbine top cover within a given time series;

[0010] The embedding representation generation module is used to perform convolution and encoding processing on preprocessed data to generate neural network embedding representations;

[0011] The encoder first processes the embedded representation based on a self-attention mechanism to obtain an attention feature matrix; then it uses pooling operations to process the attention feature matrix to generate the encoded hidden representation.

[0012] The decoder further processes the encoded hidden representation based on a self-attention mechanism to obtain the decoded hidden representation;

[0013] The neural ODE module is used to process the decoded hidden representation to obtain the predicted hidden state sequence, which is further processed by a fully connected layer to obtain the final prediction result, namely the long sequence water level of the turbine top cover of the hydropower station.

[0014] In this invention, the water level data related to the turbine top cover includes variables such as turbine unit flow rate, unit useful power, main shaft seal wear, sealing water pressure, sealing water flow rate, top cover pressure pulsation, and turbine top cover vibration data. A feature vector is constructed from the turbine top cover water level and related data (referred to as time series data) over a given time period, with a dimension of D.

[0015] The aforementioned long-sequence water level prediction system for hydropower station turbine roof based on a self-attention mechanism includes a data preprocessing module. This module preprocesses the raw water level and related data of the turbine roof for a given time series, converting them into input data required by the neural network. The data processing strategies of the preprocessing module include: (1) outlier removal and (2) missing value imputation. Based on these two data processing strategies, the data preprocessing module in this invention includes an isolated forest model and a random forest regression model.

[0016] To detect and remove outliers caused by abnormal sensor readings, this invention uses an isolated forest model. The isolated forest model is based on the conventional isolated forest algorithm already disclosed in the field and consists of multiple isolated trees (itrees), each with a binary tree structure. Its core idea is that data quickly partitioned to leaf nodes is likely to be outlier data. The construction steps of an isolated tree in the isolated forest model are as follows:

[0017] (A1) Extract n data points from the training data as training samples for the isolated tree.

[0018] (A2) Randomly select a feature from the training samples, and randomly select a value P within the range of the minimum and maximum values ​​of that feature as the root node of the current isolated tree.

[0019] (A3) Divide the samples, assigning sample points less than P to the left of the root node and sample points greater than or equal to P to the right of the root node.

[0020] (A4) Repeat steps (A2) and (A3) for the left and right parts resulting from the partition until a node contains only one sample or the maximum height of the tree is reached.

[0021] Repeat the above steps to construct N isolated trees, obtaining an isolated forest model. Then, use this model to detect outliers in the time series data. At each time point, the data traverses all isolated trees until a termination condition is met, and the path length is calculated. Under this random partitioning strategy, outliers typically have shorter paths and are identified first. Finally, the leaf node heights (i.e., path lengths) are normalized, and outliers with normalized leaf node heights less than 0.1 are removed from the time series data.

[0022] To address missing values ​​in the data, this invention uses a random forest regression model for imputation. The random forest regression model is based on the conventional random forest regression algorithm, employing different parameter choices in the construction of multiple binary decision trees, and combining multiple decision trees for final prediction. In the construction phase of the random forest regression model, bootstrap sampling is used to collect multiple different sub-training datasets from the training data to sequentially train multiple different binary decision trees. Then, the random forest regression model is used to process the time series data after outlier removal. For a specific variable value in the time series sample, the final output is obtained by averaging the prediction results of all binary decision trees. The prediction process of the binary decision trees includes the following steps:

[0023] (B1) Input a variable value of a time series sample. Starting from the root node of the binary decision tree, determine whether the current node is a leaf node. If it is, return the predicted value of the leaf node (i.e., the average value of the target variable in the current leaf sample). If not, proceed to the next step.

[0024] (B2) Based on the splitting variable and splitting value of the current node, compare the value of the corresponding variable in the sample with the splitting value of the node; if the sample variable value is greater than the current node's splitting value, then access the right child node of the current node; if the sample variable value is less than or equal to the current node's splitting value, then access the left child node of the current node.

[0025] (B3) Repeat step (B2) until a leaf node is visited, and return the predicted value of the leaf node.

[0026] Repeat steps (B1)-(B2) above to process the values ​​of each variable in the time series sample in turn. The final output data of the data preprocessing module is constructed together with the predicted values ​​of the random forest regression model and the time series data after removing outliers.

[0027] The isolated forest model and random forest regression model were used to handle outlier data and impute missing data. Furthermore, since the dimensions and ranges of different feature data in the original data vary, the data preprocessing module in this invention further includes a normalization module to normalize the different feature variable data after outlier data handling and missing data imputation, specifically calculated according to the following formula:

[0028]

[0029] In the formula, x0 and x represent the data before and after normalization, respectively.

[0030] The time series data processed by the normalization module is the final output data of the data preprocessing module, denoted as X. L .

[0031] The aforementioned long-sequence water level prediction system for the turbine roof of a hydropower station based on a self-attention mechanism includes an embedding representation generation module used to obtain the embedding representation of a neural network. In this invention, the embedding representation generation module includes a convolution processing unit, a local encoding unit, and a global encoding unit.

[0032] The convolutional processing unit is used to align the dimensions of the time-series data obtained by the data preprocessing module. Therefore, in this invention, a one-dimensional convolutional neural network is used as the convolutional processing unit to process the preprocessed time-series data to obtain the embedded representation U. L .

[0033] The local coding unit is used to add positional encoding to the preprocessed time series data samples according to their positions. Since the self-attention mechanism reduces the distance between any two positions in the sequence to a constant, positional encoding also needs to be explicitly added to the embedding representation. The purpose of positional encoding is to add sequence information to the preprocessed time series data, thus distinguishing inputs at different times. In this invention, the local coding unit uses absolute positional encoding, specifically using sine / cosine function positional encoding (PE) to encode the preprocessed time series data.

[0034] The global encoding unit is used to acquire global information from the preprocessed time series data. Since capturing long-term independence in long-term series prediction requires global information, such as hierarchical timestamps (weeks, months, and years) and agnostic timestamps (holidays, events, etc.), this invention uses a learnable, finite-dimensional global timestamp encoding SE (including hierarchical timestamps and / or agnostic timestamps) to encode the preprocessed time series data.

[0035] The specific calculation process of the above convolutional processing unit, local coding unit, and global coding unit is as follows:

[0036] U L =Conv1d(X L ;Θ conv1 (2)

[0037]

[0038] SE(pos) = W m E pos (4)

[0039] In the formula, Θ conv1 The parameters of the convolutional processing unit are represented; pos represents the position of the current variable value in the time series sample; d represents the position of the corresponding feature vector, d = 1, 2, ..., D; L represents the length of the time series, D represents the dimension of the feature vector, and W represents the position of the current variable value in the feature vector. m This represents the learnable weight parameters, whose initial values ​​are obtained by calculating the corresponding positions in the time series using local coding units, E. pos A numerical code representing a global timestamp.

[0040] Combining the three units, the embedding representation generation module finally obtains the embedding representation X. 0n as follows:

[0041] X 0n =αU L +PE (L×(d-1)+i),d +∑ p [SE (L×(d-1)+i) ] p (5)

[0042] In the formula, α is a factor balancing the relative magnitude between the scalar projection and the local or global embedding; if the time series has been standardized, then α = 1; L×(d-1)+i represents the corresponding position pos, PE (L×(d-1)+i),d This represents the matrix obtained by traversing the entire time series samples and performing position encoding, where d = 1, 2, ..., D; SE (L×(d-1)+i) This represents the matrix obtained by globally encoding the entire time series samples; p represents the number of global timestamp categories. Thus, the embedding representation X... en It combines raw information, local temporal context, and global location context information.

[0043] The aforementioned long-sequence water level prediction system for the turbine roof of a hydropower station based on a self-attention mechanism includes an encoder based on a self-attention mechanism and pooling operations. It comprises multiple stacked first multi-head attention modules (Attention(·)) and pooling modules (Pooling(·)). Each first multi-head attention module consists of multiple parallel self-attention sub-modules. The computation of each self-attention sub-module mainly consists of (query, key, value), where Q represents the query, K represents the key, and V represents the value. Q, K, and V are obtained by applying three different weight matrices to the embedding matrix X. en The multiplication results are shown in (6)-(8). Further, the attention feature vector matrix H1 is obtained through formula (9). H1 reflects the interrelationship between inputs at different positions in the time series of the embedded representation; where W is the weight matrix of the attention head. In this invention, the first multi-head attention module uses multiple different sets of W to perform multiple different mappings on the embedded representation vector to capture richer features mapped to multiple linear spaces, thus realizing multi-head attention computation; then, the computation results of multiple self-attention sub-modules are concatenated and linearly transformed to obtain the final feature matrix. This invention utilizes six self-attention sub-modules, namely six different W matrices. Furthermore, during the self-attention calculation process, a shortcut structure from the residual network is employed to mitigate the vanishing and exploding gradient problems in deep learning.

[0044] Q = W Q X en (6)

[0045] K = W K X en (7)

[0046] V = W V X en (8)

[0047]

[0048] In the formula, K T Let D' denote the transpose of K, and let D' denote the dimensions of Q, K, and V.

[0049] Furthermore, to reduce the system's time and space complexity, this invention incorporates a pooling module in the encoder's computation, replacing the original feedforward network with a pooling operation. This invention uses pooling to grant privileges to dominant features, generating a focused self-attention feature map H2 in the next layer, thereby significantly reducing the temporal dimension of the input pooling operation. The specific calculation process is as follows:

[0050]

[0051] This represents the feature matrix output by the first multi-head attention module.

[0052] To obtain deep features that have long-term dependencies, the encoder described above can set up multiple sequentially stacked first multi-head attention modules and pooling modules, so that the embedded representation of the input X... en The final encoded hidden representation H is obtained through multiple multi-head attention calculations and pooling operations. en .

[0053] The aforementioned long-sequence water level prediction system for the turbine roof of a hydropower station based on a self-attention mechanism uses a standard decoder structure, consisting of a masked multi-head attention module and a second multi-head attention module stacked together. Here, the masked multi-head attention module and the second multi-head attention module have the same structure as the first multi-head attention module described earlier.

[0054] The masked multi-head attention module is used to process the decoder input X. de Its definition is as follows:

[0055] X de =Contact(X token (X0) (11)

[0056] In the formula, Contact(·) represents channel splicing, X coken X is the starting identifier, and X0 is a placeholder for the target sequence, set to 0. Unlike the encoder, the decoder needs to mask the information of the target sequence during decoding and set the dot product of the target sequence to -∞ during attention calculation. This operation prevents each position from focusing on future positions, avoiding autoregression. A known sequence preceding the target sequence (e.g., a given time series L) is selected as the starting identifier X. token X de The input is fed into the generative decoder, and the output result H0 of the masked multi-head attention module is obtained according to formula (9).

[0057] The second multi-head attention module masks the output H0 of the multi-head attention module and the hidden state H of the encoder output. en The concatenated output is processed, and the final decoded hidden representation H is obtained through a multi-head attention computation operation. de .

[0058] The aforementioned long-sequence water level prediction system for the turbine roof of a hydropower station based on a self-attention mechanism employs a conventional structure for its neural ODE module, as disclosed in the art (see Chen RTQ, Rubanova Y, Bettencourt J, et al. Neural ordinary differential equations[C] / / Proceedings of the 32nd International Conference on Neural Information Processing Systems.2018:6572-6583). This invention uses neural ODE extrapolation to address the problem of dynamic prediction length, enabling the model to predict results of arbitrary length τ. The neural ODE module includes a hidden state sequence prediction unit and a fully connected layer. The hidden state sequence prediction unit, following formula (12), repeats this process multiple times to obtain the predicted hidden state sequence [H]. L+1 H L+2 ,…,H L+τ The hidden state sequence is input into the fully connected layer for final prediction, resulting in the predicted sequence Y. L+τ The specific process is as follows:

[0059]

[0060] Y L+τ =FFN(H L+1 ,...,H L+τ ;Θ FFN )=y L+1 ,y L+2 ,…,y L+τ (13)

[0061] In the formula, H de Let f(·) represent the hidden representation output by the decoder, f(·) represent the fully connected layer in the neural ODE, and Θ represent the hidden representation output by the decoder. f It is one of the parameters, Θ FFN It is the final predicted parameter of the fully connected layer, y L+1 ,y L+2 ,…,y L+τ This represents the τ predicted values ​​within the τ time period.

[0062] During the training process of the above-mentioned long-sequence water level prediction system for the turbine roof of a hydropower station based on the self-attention mechanism, an optimization module is further included to optimize the parameters in the system based on the loss function.

[0063] During system training, mean squared error (MSE) is used as the final loss function. The error between the output sequence and the label sequence is calculated, and gradient backpropagation is performed to optimize the system parameters. The loss function used in this invention... as follows:

[0064]

[0065]

[0066] In the formula, Θ represents all the parameters that need to be learned in the prediction system; Represents a predictive system; This represents the sequence of predicted results for the j-th training sample; Let represent the τ predicted values ​​of the j-th training sample within the τ time interval; B represents the size of the data samples for each training iteration. This represents the predicted value of the j-th training sample at time t. Let represent the true value of the j-th training sample at time t.

[0067] The system parameters are optimized according to the following formula:

[0068]

[0069] In the formula, α is the learning rate, and n represents the number of training iterations. Several iterations are performed until the prediction system performs well and stably on both the training and validation sets. At this point, a prediction system with good predictive performance is obtained.

[0070] This invention further provides a method for predicting long-sequence water levels on the turbine roof of a hydropower station based on a self-attention mechanism, which includes the following steps:

[0071] S1 preprocesses the turbine top cover water level and related data within a given time series;

[0072] S2 performs convolution and encoding on the preprocessed data to generate neural network embedding representations;

[0073] S3 first processes the embedding representation based on a self-attention mechanism to obtain an attention feature matrix; then it uses pooling operations to process the attention feature matrix to generate an encoded hidden representation.

[0074] S4 uses a self-attention mechanism to process the encoded hidden representation to obtain the decoded hidden representation;

[0075] S5 is based on the neural network ODE. It processes the decoded hidden representation to obtain the predicted hidden state sequence, and then further processes it through a fully connected layer to obtain the final prediction result, which is the long sequence water level of the turbine top cover of the hydropower station.

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] 1. This invention constructs a multivariate turbine roof water level prediction system based on a self-attention mechanism and neural ODE, using machine learning methods. This system can be used to predict future roof water levels. Compared with traditional human experience judgment and statistical methods, this system has a better modeling ability for turbine working scenarios. Moreover, compared with traditional self-attention-based neural networks, this system can better capture the correlation and long-term dependence between multiple observations, thus achieving higher water level prediction accuracy.

[0078] 2. This invention adds a pooling operation after self-attention processing, which reduces the overall computational complexity of the system and makes the prediction of long-sequence water levels of the turbine top cover faster.

[0079] 3. The prediction system of this invention introduces a neural ODE structure, which makes the time period for predicting water level variable and no longer relies on a single neural network structure.

[0080] 4. This invention can provide effective decision-making support information for the hydropower industry. It utilizes modern industrial equipment sensors to collect multi-dimensional historical observation data, including turbine unit flow rate, unit useful work, main shaft seal wear, sealing water pressure, sealing water flow rate, top cover pressure pulsation, top cover vibration, etc., to predict the top cover water level over a long period of time. This enables timely monitoring and early warning of the top cover water level, as well as analysis and monitoring of the operating efficiency of the top cover water pumps, ensuring that the top cover water level is kept within a safe and controllable range, and improving the safety of hydropower station operation. Attached Figure Description

[0081] Figure 1 This is a schematic diagram of a hydroelectric turbine in a hydroelectric power station.

[0082] Figure 2 This is a block diagram illustrating the principle of the long-sequence water level prediction system for the turbine roof of a hydropower station based on a self-attention mechanism, as described in this invention.

[0083] Terminology Explanation

[0084] Neural ODE (NODE): Neural ordinary differential equation. In conventional neural networks, state transitions are obtained through a series of discrete transformations; however, these models struggle to describe events occurring at continuous and irregular intervals. In such cases, neural ODEs are better suited to describe these dynamic systems; they are equations with learnable parameters. NODEs consider an infinite number of hidden state updates in a neural network, replacing the discrete sequence transformations of the hidden states. Detailed Implementation

[0085] The present invention will be further described with reference to the accompanying drawings.

[0086] Example 1

[0087] The following is combined with Figure 2 This embodiment provides a detailed explanation of the long-sequence water level prediction system for the turbine roof of a hydropower station based on a self-attention mechanism.

[0088] This embodiment provides a long-sequence water level prediction system for hydropower station turbine roofs based on a self-attention mechanism. It includes a data preprocessing module, an embedded representation generation module, an encoder, a decoder, and a neural ODE module. In this embodiment, the input data for long-sequence water level prediction of hydropower station turbine roofs includes the turbine roof water level and related data within a given time series. The related data includes turbine unit flow rate, unit useful power, main shaft seal wear, sealing water pressure, sealing water flow rate, roof pressure pulsation, and turbine roof vibration data variables. A feature vector with dimension D is constructed from the turbine roof water level and related data.

[0089] The aforementioned data preprocessing module is used to preprocess the turbine top cover water level and related data within a given time series. In this embodiment, the data preprocessing module includes an isolated forest model for removing outliers, a random forest regression model for imputing missing data, and a normalization module. The isolated forest model consists of N (N=100) isolated trees. Each isolated tree has a binary tree structure. The random forest regression model consists of M (M=60) binary decision trees. The normalization module performs normalization processing on different feature variable data after outlier processing and missing data imputation, specifically calculated according to the following formula:

[0090]

[0091] In the formula, x0 and x represent the data before and after normalization, respectively.

[0092] The aforementioned embedding representation generation module is used to perform convolution and encoding processing on preprocessed data to generate neural network embedding representations. In this embodiment, the embedding representation generation module includes a convolution processing unit, a local encoding unit, and a global encoding unit. In this embodiment, a one-dimensional convolutional neural network U is used as the convolution processing unit, with a kernel size of 3, a stride of 1, edge padding of 2, and a kernel count of 1. The local encoding unit uses sine and cosine function positional encoding PE. The global encoding unit uses learnable, finite-dimensional global timestamp encoding SE.

[0093] The specific calculation process of the above convolutional processing unit, local coding unit, and global coding unit is as follows:

[0094] U L =Conv1d(X L ;Θ conv1 (2)

[0095]

[0096] SE(pos) = W m E pos (4)

[0097] In the formula, Θ conv1 The parameters of the one-dimensional convolutional neural network are: pos represents the position of the current variable value in the time series sample; d represents the position of the corresponding feature vector, d = 1, 2, ..., D; L represents the length of the time series, D represents the dimension of the feature vector, and W... m This represents the learnable weight parameters, whose initial values ​​are obtained by calculating the corresponding positions in the time series using local coding units, E. pos A numerical code representing a global timestamp. This numerical code consists of symbols in a fixed format, for example, Wednesday can be represented as 0307, ​​17:00 can be represented as 1724, etc.

[0098] Combining the three units, the embedding representation generation module finally obtains the embedding representation X. en as follows:

[0099] X en =αU L +PE (L×(d-1)+i),d +∑ p [SE (L×(d-1)+i) ] p (5)

[0100] In the formula, α is a factor that balances the relative magnitude between the scalar projection and the local or global embedding. If the time series has been standardized, then α = 1; = ×(d-1) + i represents the corresponding position pos, PE (L×(d-1)+i),d This represents the matrix obtained by traversing the entire time series samples and performing position encoding, where d = 1, 2, ..., D; SE (L×(d-1)+i) This represents the matrix obtained by traversing the entire time series sample for global encoding; p represents the number of types of global timestamps. In this embodiment, since only hierarchical timestamps are considered, p = 3, including three types: week, month, and year.

[0101] The encoder described above first processes the embedded representation based on a self-attention mechanism to obtain an attention feature vector matrix; then, it uses a pooling operation to process the attention feature matrix to generate an encoded hidden representation. In this embodiment, it consists of two stacked sets of first multi-head attention modules and pooling modules.

[0102] The first multi-head attention module consists of six parallel self-attention sub-modules. The computation of each self-attention sub-module primarily comprises (query, key, value). Here, Q represents the query, K represents the key, and V represents the value. Q, K, and M are calculated using three different weight matrices and the embedding matrix X. en The product is obtained by multiplication (as shown in (6)-(8)). Further, the attention feature vector matrix H1 is obtained through formula (9). H1 reflects the relationship between inputs at different positions in the time series of the embedded representation; where W is the weight matrix of the attention head.

[0103] Q = W Q X en (6)

[0104] K = W K X en (7)

[0105] V = W V X en (8)

[0106]

[0107] In the formula, K T Let D' denote the transpose of K, and let D' denote the dimensions of Q, K, and V.

[0108] The first multi-head attention module utilizes six different sets of W to perform multiple different mappings on the embedded representation vectors, thereby capturing richer features mapped to multiple linear spaces and realizing multi-head attention computation. Then, the computation results from multiple self-attention sub-modules are concatenated and linearly transformed to obtain the final feature matrix. These represent the outputs of the six self-attention submodules.

[0109] In this embodiment, the pooling module first processes the obtained feature matrix. One-dimensional convolution is performed, followed by ReLU processing, and then max pooling is used to obtain the hidden representation of the pooling module. The specific calculation process is as follows:

[0110]

[0111] The obtained hidden representation H2 is used as input, and then processed sequentially by the first multi-head attention module and the pooling module to complete the processing of the embedded representation input X. en The final encoded hidden representation N is obtained through two multi-head attention calculations and pooling operations. en .

[0112] The decoder described above further processes the encoded hidden representation based on a self-attention mechanism to obtain the decoded hidden representation. In this embodiment, the decoder is composed of a masking multi-head attention module and a second multi-head attention module stacked together. Here, the masking multi-head attention module and the second multi-head attention module have the same structure as the first multi-head attention module given above.

[0113] The masked multi-head attention module is used to process the decoder input X. de Its definition is as follows:

[0114] X de =Contact(X token (X0) (11)

[0115] In the formula, Contact(·) represents channel splicing, X token X is the start identifier, and X0 is a placeholder for the target sequence, set to 0. Unlike the encoder, the decoder needs to mask the information of the target sequence during decoding, and the dot product result of the target sequence needs to be set to -∞ during attention calculation. A known sequence preceding the target sequence is selected as the start identifier X. token X de The input is fed into the generative decoder, and the output result H0 of the masked multi-head attention module is obtained according to formula (9).

[0116] The second multi-head attention module masks the difference between the multi-head attention output H0 and the hidden state H of the encoder output. en The concatenated output is used as input, and then the final decoded hidden representation H is obtained through a multi-head attention computation operation. de .

[0117] The aforementioned neural ODE module is used to process the decoded hidden representation to obtain the predicted hidden state sequence, which is further processed by a fully connected layer to obtain the final prediction result, namely the long sequence water level of the turbine top cover of the hydropower station.

[0118] The neural ODE module consists of a hidden state sequence prediction unit and fully connected layers. The hidden state sequence prediction unit obtains the predicted hidden sequence at time τ according to the following formula:

[0119]

[0120] In the formula, f(·) represents the fully connected layer in the neural ODE, and Θ f It is one of the parameters, H de This represents the hidden representation output by the decoder.

[0121] By repeating this process multiple times according to the above formula (12), the predicted hidden state sequence [H] can be obtained. L+1 H L+2,…,H L+τ ].

[0122] The hidden state sequence [H] L+1 H L+2 ,…,H L+τ The input is processed by a fully connected layer to perform the final prediction, resulting in the predicted sequence Y of the turbine top cover water level length at time τ. L+τ :

[0123] Y L+τ =FFN(H L+1 ,...,H L+τ ;Θ FFN )=[y L+1 ,y L+2 ,…,y L+τ (13)

[0124] In the formula, Θ FFN It is the final predicted parameter of the fully connected layer, y L+1 ,y L+2 ,…,y L+τ This represents the τ predicted values ​​within the τ time period.

[0125] Example 2

[0126] This embodiment is a further improvement based on Embodiment 1.

[0127] During the training process of the long sequence water level prediction system for the turbine roof of a hydropower station based on the self-attention mechanism provided in Example 1, in order to optimize the system parameters, the prediction system further includes an optimization module for optimizing the parameters in the system based on the loss function.

[0128] Loss function used during system training as follows:

[0129]

[0130]

[0131] In the formula, Θ represents all the parameters that need to be learned in the prediction system; Represents a predictive system; This represents the sequence of predicted results for the j-th training sample; Let represent the τ predicted values ​​of the j-th training sample within the τ time interval; B represents the size of the data samples for each training iteration. This represents the predicted value of the j-th training sample at time t. Let represent the true value of the j-th training sample at time t.

[0132] The system parameters are optimized according to the following formula:

[0133]

[0134] In the formula, α is the learning rate, and n represents the number of training iterations. Several iterations are performed until the prediction system performs well and stably on both the training and validation sets. At this point, a prediction system with good predictive performance is obtained.

[0135] Application examples

[0136] This application example uses the long-sequence water level prediction system for the turbine roof of a hydropower station based on the self-attention mechanism provided in Example 2 to train and test on the real dataset of the Dagangshan Hydropower Station on the Dadu River, in order to illustrate the effect of the long-sequence water level prediction system and method for the turbine roof of a hydropower station based on the self-attention mechanism provided in this invention.

[0137] To meet the requirements of deep learning model learning problems, this application example divides the dataset D required for each test into a training set D in a 3:1:1 ratio. train Validation set D val and test set D test In each dataset, X L ={x1,...,x L |x i ∈R D} represents the multidimensional time-series data of the caprock water level, where L represents the input sequence length and D represents the dimension of the data features. The training and validation sets are used to train the system parameters and tune hyperparameters, while the test set is used to test the performance. Furthermore, to verify the effectiveness of predicting the caprock water level at different future time periods, this application example sets various values ​​for the encoder input length and prediction length. Considering the significance of time, this application example takes two future time periods (denoted by τ) of 60 minutes and 120 minutes in the future for prediction and reports the results.

[0138] (I) System Training

[0139] Train the system using the training set data by following these steps:

[0140] S1 uses a data preprocessing module to preprocess the turbine top cover water level and related data (referred to as time series data) in the time series of the training set and validation set.

[0141] This mainly involves removing abnormal data and interpolating missing data in the time series data collected by the sensor.

[0142] (1) The isolated forest model was used to detect and remove time series data.

[0143] The steps for constructing an isolated tree in an isolated forest model are as follows:

[0144] (A1) Extract n data points from the training set as training samples for the isolated tree.

[0145] (A2) Randomly select a feature from the training samples, and randomly select a value P within the range of the minimum and maximum values ​​of that feature as the root node of the current isolated tree.

[0146] (A3) Divide the samples, assigning sample points less than P to the left of the root node and sample points greater than or equal to P to the right of the root node.

[0147] (A4) Repeat steps (A2) and (A3) for the left and right parts resulting from the partition until a node contains only one sample or the maximum height of the tree is reached.

[0148] Repeat the above steps to construct N isolated trees, obtaining an isolated forest model. Then, use this model to detect outliers in the time series data, traversing all isolated trees at each time point until a termination condition is met, and calculating the path length. Finally, normalize the leaf node height data (i.e., path length), removing outliers with normalized leaf node heights less than 0.1 from the time series data.

[0149] (2) Use a random forest regression model to impute missing values.

[0150] Bootstrap sampling is used to collect multiple different sub-training datasets from the training set data to train M different binary decision trees in sequence. For specific training methods, please refer to Breiman L. Random forests[J].Machinelearning,2001,45(1):5-32.

[0151] This section focuses on time series data after removing outliers. The prediction process of a binary decision tree includes the following steps:

[0152] (B1) Input a variable value of a time series sample. Starting from the root node of the binary decision tree, determine whether the current node is a leaf node. If it is, return the predicted value of the leaf node (i.e., the average value of the target variable in the current leaf sample). If not, proceed to the next step.

[0153] (B2) Based on the splitting variable and splitting value of the current node, compare the value of the corresponding variable in the sample with the splitting value of the node; if the sample variable value is greater than the current node's splitting value, then access the right child node of the current node; if the sample variable value is less than or equal to the current node's splitting value, then access the left child node of the current node.

[0154] (B3) Repeat step (B2) until a leaf node is visited, and return the predicted value of the leaf node.

[0155] For all binary decision trees, repeat steps (B1)-(B2) above, and take the average of the prediction results of all binary decision trees as the final output. A random forest regression model is used to process the variable values ​​of the time series samples after removing outliers. The predicted values ​​of the random forest regression model and the time series data after removing outliers are further processed by the normalization module below to form the final output data of the data preprocessing module.

[0156] (3) The normalization module is used to normalize the different feature variable data after outlier processing and missing data imputation. The specific calculation formula is as follows:

[0157]

[0158] In the formula, x0 and x represent the data before and after normalization, respectively.

[0159] The time series data processed by the normalization module is the final output data of the data preprocessing module, denoted as X. L .

[0160] S2 uses the embedding representation generation module to perform convolution and encoding on the preprocessed data to generate neural network embedding representations.

[0161] The normalized training set data obtained in step S1 is processed using convolutional processing units, local encoding units, and global encoding units according to formulas (2)-(4), and the final embedding representation X is obtained according to formula (5). en .

[0162] S3 utilizes an encoder to first process the embedded representation based on a self-attention mechanism to obtain an attention feature vector matrix; then, it uses a pooling operation to process the attention feature vector matrix to generate an encoded hidden representation.

[0163] According to formulas (6)-(10), the embedded representation X is sequentially processed using two sets of first multi-head attention modules and pooling modules. en The final encoded hidden representation H is obtained by performing two multi-head attention calculations and pooling operations. en .

[0164] S4 uses a decoder, based on a self-attention mechanism, to process the encoded hidden representation to obtain the decoded hidden representation.

[0165] First, construct the input X of the decoder according to formula (11). de Then, the masked multi-head attention module is used to process the decoder input X in the same way as the first multi-head attention module. deThe process is performed to obtain the output result H0 of the masking multi-head attention module; the output result H0 of the masking multi-head attention module is then compared with the hidden state H output by the encoder. en The concatenation is used as input to the second multi-head attention module, and further processed according to the same operations as the first multi-head attention module to obtain the final decoded hidden representation H. de .

[0166] S5 utilizes the neural ODE module, based on the neural network ODE, to process the decoded hidden representation to obtain the predicted hidden state sequence, and further processes it through a fully connected layer to obtain the final prediction result, namely the long sequence water level of the turbine top cover of the hydropower station.

[0167] This step first involves repeating the process multiple times according to formula (12) to obtain the predicted hidden state sequence [H] within the τ time period. L+1 H L+2 ,…,H L+τ ].

[0168] The hidden state sequence [H] L+1 H L+2 ,…,H L+τ The input is fully connected layer for final prediction (according to formula (13)), and the predicted result sequence Y of the turbine top cover water level length at time τ is obtained. L+τ .

[0169] S6 optimizes system parameters based on the system loss function.

[0170] The system loss function is obtained according to formula (14), and then the system parameters are optimized according to formula (16).

[0171] S7 Then, using the parameter-optimized system, perform validation on the validation set by referring to steps S2-S5 above.

[0172] Then repeat steps S2-S7 for several iterations until the prediction system performs well and stably on both the training and validation sets.

[0173] (II) System Testing

[0174] Using the trained system, refer to steps S1-S5 above. First, use the data preprocessing module to process the test set data, and then use the processed test set data to test the system.

[0175] This application example uses four baselines to compare and evaluate prediction performance: ARIMA, LSTM, GRU, and Transformer.

[0176] This application example uses the mean squared error (MSE) and mean absolute error (MAE) of the top cover water level as evaluation indicators (the lower the value, the better the model method). Ours represents the results of the model of this patented invention, with the best results indicated in bold.

[0177] Table 1: Prediction performance on the application example dataset

[0178]

[0179]

[0180] The remaining methods in the table are described below:

[0181] ARIMA: One of the most common statistical models for time series forecasting based on historical data. For implementation details, please refer to the literature: [Wang, Wen-chuan, et al. "Improving forecasting accuracy of annual runofftime series using ARIMA based on EEMD decomposition." Water Resources Management 29.8(2015):2655-2675.]

[0182] LSTM: A variant of RNN, it introduces a gating mechanism based on RNN, consisting of an input gate, a forget gate, and an output gate. It is widely used for the extraction and mining of time series information. For implementation details, please refer to the literature: [Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural computation,1997,9(8):1735-1780.]

[0183] GRU is an improvement on LSTM, merging the forget gate and input gate of LSTM into a single update gate. This model only contains update and reset gates, reducing computational cost and memory usage. For implementation details, please refer to [Apaydin, Halit, et al. "Comparative analysis of recurrent neural network architectures for reservoir inflow forecasting." Water 12.5(2020):1500.]

[0184] Transformer: Replaces the RNN network structure with a self-attention mechanism to align and capture temporal information. It utilizes a layered self-attention mechanism to better capture the most important parts of multiple input sequences. For implementation details, please refer to the literature: [Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C] / / Advances in neural information processing systems.2017:5998-6008.]

[0185] As can be seen from the experimental results in Table 1, the method for predicting the long sequence water level of the turbine top cover of a hydropower station based on the self-attention mechanism of this invention can be used for the task of predicting the water level of the turbine top cover. Compared with other machine learning algorithms, it has a significant improvement in both mean square error and mean absolute error.

[0186] Therefore, it can be seen that the present invention utilizes a long-sequence water level prediction system for the turbine roof of a hydropower station based on a self-attention mechanism. By inputting historical multidimensional observation values, it can predict the water level of the turbine roof over a relatively long period of time in the future with relatively high accuracy.

[0187] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A long-sequence water level prediction system for the turbine roof of a hydropower station based on a self-attention mechanism, characterized in that, include: The data preprocessing module is used to preprocess the turbine top cover water level and related data within a given time series; An embedding representation generation module is used to perform convolution and encoding processing on preprocessed data to generate neural network embedding representations; the embedding representation generation module includes a convolution processing unit, a local encoding unit, and a global encoding unit; The encoder first processes the embedded representation based on a self-attention mechanism to obtain an attention feature matrix; then it processes the attention feature matrix using a pooling operation to generate an encoded hidden representation; the encoder includes multiple stacked first multi-head attention modules and pooling modules; the first multi-head attention module is composed of multiple parallel self-attention sub-modules; The decoder further processes the encoded hidden representation based on a self-attention mechanism to obtain the decoded hidden representation; The decoder is composed of a stacked masked multi-head attention module and a second multi-head attention module; the masked multi-head attention module and the second multi-head attention module have the same structure as the first multi-head attention module described above; the masked multi-head attention module is used to process the decoder input. Its definition is as follows: ； In the formula, Indicates channel splicing. It is the starting identifier. These are placeholders for the target sequence; The input is fed into the generative decoder to obtain the output of the masked multi-head attention module. ; A second multi-head attention module is used to mask the output of multi-head attention. Hidden state of encoder output The concatenated output is used as input, and then the final decoded hidden representation is obtained through a multi-head attention computation operation. ; The neural ODE module is used to process the decoded hidden representation to obtain the predicted hidden state sequence, which is further processed by a fully connected layer to obtain the final prediction result, namely the long sequence water level of the turbine top cover of the hydropower station.

2. The long-sequence water level prediction system for the turbine roof of a hydropower station based on a self-attention mechanism according to claim 1, characterized in that, The data preprocessing module includes an isolated forest model for removing outliers, a random forest regression model for imputing missing data, and a normalization module.

3. The long-sequence water level prediction system for the turbine roof of a hydropower station based on a self-attention mechanism according to claim 1, characterized in that, The specific calculation process of the convolutional processing unit, local coding unit, and global coding unit is as follows: （2）； （3）； （4）； In the formula, Indicates the parameters of the convolution processing unit; This indicates the position of the current variable value within the time series sample; This represents the position in the feature vector corresponding to the current variable value, d=1,2,…,D; L represents the length of the time series, and D represents the dimension of the feature vector. The weight parameters represent learnable parameters, whose initial values ​​are obtained by calculating the corresponding positions in the time series using local coding units. Numerical encoding representing a global timestamp; （5）； In the formula, It is a factor that balances the relative magnitudes of the scalar projection and the local or global embedding, if the time series has been normalized. ; This represents the corresponding position pos. This represents the matrix obtained by traversing the entire time series sample and performing position encoding, where d = 1, 2, …, D; This represents the matrix obtained by globally encoding the entire time series sample. Indicates the number of types of global timestamps.

4. The long-sequence water level prediction system for the turbine roof of a hydropower station based on a self-attention mechanism according to claim 1, characterized in that, The computation of the self-attention submodule is mainly composed of Composition; among which, This indicates a query. Indicates key, This represents the value. , , It is achieved through three different weight matrices and embedding matrices. The product obtained by multiplication is shown in (6)-(8); （6）； （7）； （8）； The attention feature vector matrix is ​​further obtained through formula (9). : （9）； In the formula, express transpose, Represents the dimensions of Q, K, and V.

5. The long-sequence water level prediction system for the turbine roof of a hydropower station based on a self-attention mechanism according to claim 4, characterized in that, The pooling module obtains the feature matrix from the first multi-head attention module. One-dimensional convolution is performed, followed by ReLU processing, and then max pooling is used to obtain the hidden representation of the pooling module. The specific calculation process is as follows: （10）。 6. The long-sequence water level prediction system for the turbine roof of a hydropower station based on a self-attention mechanism according to claim 1, characterized in that, The neural ODE module includes a hidden state sequence prediction unit and a fully connected layer; the hidden state sequence prediction unit, following formula (12), repeats this process multiple times to obtain the predicted hidden state sequence. The hidden state sequence is input into the fully connected layer for final prediction, resulting in a prediction sequence. The specific process is as follows: （12）； （13）； In the formula, This represents the hidden representation output by the decoder. This represents the fully connected layer in a neural ODE. These are some of the parameters. These are the parameters for the final predicted fully connected layer. express within the time period One predicted value.

7. A method for predicting long-sequence water level of the turbine roof of a hydropower station based on a self-attention mechanism, characterized in that, Using the system according to any one of claims 1 to 6, the following steps are performed: S1 preprocesses the turbine top cover water level and related data within a given time series; S2 performs convolution and encoding on the preprocessed data to generate neural network embedding representations; S3 first processes the embedding representation based on the self-attention mechanism to obtain the attention feature matrix; then it uses pooling operation to process the attention feature matrix to generate the encoded hidden representation. S4 uses a self-attention mechanism to process the encoded hidden representation to obtain the decoded hidden representation; S5 is based on the neural network ODE. It processes the decoded hidden representation to obtain the predicted hidden state sequence, and then further processes it through a fully connected layer to obtain the final prediction result, which is the long sequence water level of the turbine top cover of the hydropower station.

Images