Water quality prediction method based on transformer sparse attention mechanism

Abstract

The application provides a water quality prediction method based on a transformer sparse attention mechanism, including the following steps: obtaining and storing water quality data through a collection site to construct a water quality data set; missing water quality data is filled in by a multiple interpolation method, the overall data set is normalized by mean value, and the data set is divided into a training set and a test set; the training set data is input into a time series decomposition module to extract time series features; the data is input into an Embedding layer to add time position features to obtain a sparse matrix X1, and the feature vector obtained by processing the sparse matrix X1 by an encoder is input into a decoder; the data is input into an Embedding layer to add time position features to obtain a sparse matrix X2, and the decoder processes the data input by the encoder and the sparse matrix X2 to obtain prediction data; the model is trained and iteratively updated to obtain predicted water quality data.

Description

Technical Field

[0001] This invention belongs to the field of water quality prediction in ecological and environmental protection, and designs a water quality prediction method based on the Transformer sparse attention mechanism. Background Technology

[0002] The quality of the water environment affects human life and the development of the ecological environment, impacting the long-term development of society. With rapid industrial development and accelerated urbanization, water issues have become more complex, and problems such as water pollution, water shortages, and water quality deterioration are constantly emerging. Traditional water pollution control mainly relies on setting up monitoring stations, collecting water quality data, and uploading it to terminal equipment. This approach cannot predict future water quality changes or provide timely responses. Water quality prediction is a future trend in the industry and a crucial step in water environment prevention. Therefore, improving the accuracy and transferability of water quality prediction models has become an urgent and important issue to be addressed.

[0003] Traditional water quality prediction methods have several shortcomings. For example, mechanistic models rely heavily on human experience for their structure and parameters, resulting in high complexity and difficulty in dynamic calibration. In river water bodies that change dynamically over time, it's necessary to select the appropriate mechanistic model based on the specific scenario. When there are many influencing factors and complex mechanisms, problems such as difficulty in model establishment, acquisition of relevant parameters, or insufficient water quality monitoring data may arise. With the continuous development of artificial intelligence, methods such as RNN, LSTM, and SVM are being used for simulation in water quality prediction. However, the selection and optimization of hyperparameters in these methods mainly rely on subjective experience and continuous trial and error, without considering the differences in the contribution intensity of other water quality parameters to the prediction parameters. This severely restricts the accuracy and generalization performance of predictions. In recent years, the continuous development of hardware has provided new solutions for long-term time-series predictions and model transferability. More and more people are applying Transformers to the prediction field. However, Transformer models are mostly used in natural language processing, and their accuracy and generalization in water quality prediction still need to be improved. Summary of the Invention

[0004] To address the technical problems mentioned in the background section, this invention provides a water quality prediction method based on the Transformer sparse attention mechanism. The technical means employed in this invention are as follows:

[0005] The water quality prediction method based on the transformer sparse attention mechanism includes the following steps:

[0006] Step 1: Construct a water quality data input sequence based on the water quality data and information obtained from the tested water area by the acquisition equipment;

[0007] Step 2: When data transmission is abnormal, delete the abnormal data and fill in the missing data using multiple interpolation. Then, normalize the mean of the filled data and divide the normalized data into training and test sets in an 8:2 ratio. Use the training set as the input data for the model.

[0008] Step 3: After position encoding the training set data output by the time series module, input the data into the Input Embedding to perform position encoding and obtain the sparse matrix X1.

[0009] Step 4: Input the training set time series data obtained in Step 2 into the Output Embedding layer for position encoding to obtain a sparse matrix X2. Input the sparse matrix X2 into the first Mask sparse attention module of the Decoder to calculate and extract the feature matrix to obtain the query vector. The query vector is then passed through a residual normalization layer and input into the next sparse attention module. The processed query vector, along with the output key vector and value vector of the last layer of the encoder, is input into the second sparse attention mechanism layer to obtain the output sequence. The obtained output sequence is then passed through a residual connection and normalization layer and input into a feedforward neural network to extract the feature vector. Another residual connection and normalization is performed to obtain the result vector. The decoder unit repeats the calculation 7 times, and the last decoder unit outputs the data feature vector. The score obtained by passing the data feature vector through the Softmax layer is converted into the probability of the future water quality data trend, and the value with the highest probability is selected as the final output water quality prediction value.

[0010] Step 5: Input the training set data and iteratively update the training according to the loss function until the model has been trained for the preset number of iterations. After completing the last iteration, terminate the training. Select the model with the smallest absolute error and mean standard error among the trained models and save the model structure. Input the test set data into the saved model structure to verify the effect of the saved model.

[0011] Compared with the prior art, the present invention has the following advantages:

[0012] This invention makes predictions based on raw water quality data, forecasting potential future water pollution and thus reducing loss of life and property caused by natural factors.

[0013] This invention utilizes localized data, adapting to local conditions. The collected data comes from local environmental monitoring stations, ensuring timeliness. All parameter options can be adjusted according to local water bodies, making localized deployment highly valuable.

[0014] The model of this invention is transferable and shows good predictive performance in different water areas and for predicting different indicators, making the model applicable to multiple water areas.

[0015] This invention solves the problem of long-term time-series prediction, reduces computational time complexity while improving prediction accuracy, saves computational resources, reduces the cost of model deployment, and improves economic efficiency. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the steps of the water quality prediction method based on the Transformer sparse attention mechanism of this invention.

[0018] Figure 2 This is a flowchart illustrating the steps of the water quality prediction method based on the Transformer sparse attention mechanism of this invention.

[0019] Figure 3 This is a sample prediction image of the water quality prediction method based on the Transformer sparse attention mechanism of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] like Figure 1 As shown, this invention provides a water quality prediction method based on the Transformer sparse attention mechanism, specifically including the following steps:

[0023] Step 1: Collect data in the designated water area by deploying front-end water quality monitoring equipment. The collected data is stored in the database as basic information data. The basic information data obtained includes, but is not limited to, water quality information such as COD, BOD, total phosphorus, total nitrogen, and ammonia nitrogen, and is stored in the database for easy retrieval.

[0024] Step 2: When blank data or data transmission anomalies occur, delete the abnormal data, and fill in the missing data using multiple interpolation. After data filling, normalize the mean of all data, and then divide the normalized data into training and test sets as input data for the model. Finally, process the time series data using a moving average filter. The specific process of Step 2 is as follows:

[0025] Step 2.1: However, if the front-end acquisition device transmits blank data or the data transmission is abnormal, delete the abnormal data.

[0026] Step 2.2: Missing data are imputed using multiple interpolation. The imputed data is then normalized to the mean and arranged chronologically as time series data as input to the model. The mean normalization formula is:

[0027]

[0028] In the formula, Take the average of all values, where value is the value taken at the current moment.

[0029] Step 2.3: Divide the mean-normalized data into training and test sets according to different months. The training set is used as the training data for the model, and the test set is used to verify the model's accuracy. The ratio of the training set to the test set is 8:2.

[0030] Step 2.4: To capture long-term trends and cyclical characteristics, the time data series is processed using a moving average filter:

[0031] Trend information T is obtained from input data of length d. t The formula is as follows:

[0032]

[0033] The formula for periodic decomposition is as follows.

[0034]

[0035] The final processed time data sequence X is obtained. t =T t +S t +R t .

[0036] In the formula n i Let n be the number of occurrences of seasoni. d D t The total number is equal to n-2q, S i For the trend term, R t For noise, X i The average value of the data of length i is taken. It is the average of the sum of all D values.

[0037] Step 3: Input the temporally decomposed sequence into the Input Embedding layer. The Embedding layer adds the positional encoding information at each time step to the corresponding temporal position embedding vector to obtain the data representation matrix vector, thus obtaining the encoder sparse matrix X1. Set the Encoder input layer to 7. For example... Figure 2 As shown, the obtained sparse matrix X1 is input into the encoder to extract temporal features. After calculation by the encoder unit, the encoded information matrix of all water quality data in the sequence can be obtained, where the matrix dimension of the encoder unit output matrix is ​​consistent with the dimension of the input matrix. The encoder unit includes a sparse attention module (ProbAttention), two residual normalization layers (Add & layernorm), and a feedforward neural network layer. Multiple sparse attention mechanisms are concatenated together to form the sparse attention module. The sparse attention coefficients are: Where d is the matrix dimension, i.e., the number of columns in the matrix, and Q represents the query vector. The sparse matrix is ​​obtained as a sparsity measure, where K represents the key matrix and V represents the value matrix. The residual connection layer is used to address the training problem of multi-layer networks, enabling rapid network convergence. Finally, the encoder's output matrix is ​​obtained through a feedforward neural network and a residual normalization layer. The specific steps in step 3 are as follows:

[0038] Step 3.1: Input the time-series decomposed sequence into the InputEmbedding layer. The Embedding layer adds the location encoding information at each time step to the corresponding temporal location embedding vector to obtain the data representation matrix, thus obtaining the encoder sparse matrix X1. The embedding vector of the water quality data location information is represented as PE, and the formula for PE is...

[0039]

[0040]

[0041] Where pos represents the relative position of the time-series location information to be encoded in the water quality data time series, i represents the dimension of the time series vector, and d represents the vector dimension in the global data.

[0042] The sparse matrix X1 obtained in 3.2 is input into the attention mechanism module and the score is extracted through sparse attention calculation, thereby obtaining the output vector from the global data.

[0043] The sparse attention calculation method is as follows:

[0044] Step 3.2.1: For each query vector q i All randomly sampled quantitative key vectors K

[0045] Step 3.2.2: Calculate the attention M(q) for each query vector. i K), its formula is as follows:

[0046]

[0047] Step 3.2.3: Select m specific valid query vectors q from the sparse attention method. i The method for determining m is as follows:

[0048] m = c ln L;

[0049] Step 3.2.4: Take action Calculate the attention score.

[0050] Step 3.3: The data obtained in Step 3.2 is processed through a residual normalization layer to perform nonlinear processing on the data to address the problems of gradient vanishing and network degradation, and to standardize the data. The output value is then passed to the feedforward neural network layer. The residual connection layer is represented as follows:

[0051]

[0052] Where E[X] represents the expected value, Var[x] represents the variance, and γ and β represent the learnable parameters.

[0053] Step 3.4: Extract features from the output vector of Step 3.3 through a feedforward neural network layer.

[0054] Step 3.5: Pass the output vector obtained in step 3.4 to the residual normalization layer for processing, and then pass it to the next layer decoder.

[0055] Step 3.6: Repeat steps 3.2-3.5 until the obtained water quality data time series training set is obtained. Step 4: Input the training set time series data obtained in step 2 into the Output Embedding layer for processing to obtain the sparse matrix X2. Input the encoder output vector into the decoder (Dedcoder). The number of decoder layers is set to 7 (each decoder layer includes a Masked Sparse Attention module, a Sparse Attention module, a Feedforward Neural Network layer, and three Residual Connection layers. The first Masked Sparse Attention mechanism layer is used to record information between the current inputs, and the second Sparse Attention mechanism layer is used to predict the output). The specific steps of step 4 are as follows:

[0056] Step 4.1: Input the training set time series data obtained in Step 2 into the Output Embedding layer to obtain the sparse matrix X2;

[0057] Step 4.2: The sparse matrix X2 obtained in Step 4.1 after position encoding is input into the sparse attention module. The input sparse matrix is ​​aligned, and the query vector is output. After passing through the residual normalization layer, it is input into the next sparse attention module.

[0058] Step 4.3: The key vector and value vector of the encoder's last layer output from Step 4.2 are input into the second sparse attention mechanism layer to obtain the output sequence;

[0059] Step 4.4: The result obtained in step 4.3 is input into the residual normalization layer to obtain the residual normalized connection value;

[0060] Step 4.5: Extract feature vectors from the normalized connection values ​​obtained in Step 4.4 through a feedforward neural network, and then perform residual connection and normalization again to obtain the result vector;

[0061] Step 4.6: Repeat steps 4.2-4.5 above 7 times, and the last decoder outputs the data feature vector;

[0062] Step 4.7: Convert the scores obtained from the data feature vectors through the Softmax layer into probabilities of future water quality trends, and select the value with the highest probability as the final output water quality prediction value.

[0063] Step 5: Iteratively update the model's loss function during training, repeating steps 3 and 4. Training stops after the preset number of iterations is reached and the final iteration is completed. Select the time-series prediction model with the smallest absolute error (MAE) and mean standard error (MSE) among the trained models and save its structure. Input the test set data into the model to verify its performance. During model training, a continuously decreasing learning rate is selected based on the number of iterations. The loss function formula is as follows:

[0064]

[0065] The learning rate is calculated using the following formula:

[0066] learn (epoch) =lrate (epoch-1) ×0.96 epach ;

[0067] Where learn is the current learning rate, lrate is the learning rate of the previous round, and epoch is the number of iterations.

[0068] Calculate the mean absolute error and mean standard error evaluation index values ​​for prediction results with different prediction step sizes:

[0069]

[0070] The results show that the evaluation scores are significantly improved compared to the traditional Attention mechanism of Transformer.

[0071] Water source one mae mse This method 0.3074 0.1914 Transformer Traditional Attention 0.4216 0.3266 Water source 2 mae mse This method 0.2606 0.5105 transformer traditional attention 0.3266 0.5715

[0072] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content in the several embodiments provided in this application can be implemented in other ways.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A water quality prediction method based on a transformer sparse attention mechanism, characterized in that, The method comprises the following steps: Step 1, constructing a water quality data input sequence by water quality data and information obtained from a detection water area according to a collection device; Step 2, deleting abnormal data when data transmission is abnormal, filling in the missing data by a multiple interpolation method, then performing mean value normalization processing on the filled-in data, and dividing the normalized data into a training set and a test set according to a ratio of 8:2; taking the training set as input data of a model; Step 3, performing position coding on the training set data output by a time sequence module, inputting the data into an InputEmbedding embedding to obtain a sparse matrix X1 through position coding; the time sequence module is processed by a moving average filtering to obtain periodic characteristics of the water quality data; the sequence processed by the moving average filtering is input into an Embedding layer, and position coding information of each time and an embedding vector corresponding to the time are added to obtain a sparse matrix; the time position coding is obtained by the following formula, and the embedding vector of the water quality data position information is represented as PE, and the PE formula is: ； ； wherein pos represents the relative position of the time series position information to be encoded in the time series of water quality data, i denotes the dimension of the time series vector, d denotes the vector dimension in the global data; Step 4, inputting the training set time sequence data obtained in step 2 into an Output Embedding layer to perform position coding to obtain a sparse matrix X2, inputting the sparse matrix X2 into a first Mask sparse attention module of a Decoder decoder to calculate and extract a feature matrix to obtain a query vector; the query vector is input into a next layer sparse attention module through a residual normalization layer, and the processed query vector and an output key vector and a value vector of the last layer of the encoder are input into a second layer sparse attention mechanism layer to obtain an output sequence; the obtained output sequence is input into a feedforward neural network after residual connection and normalization to extract a feature vector, and a result vector is obtained after residual connection and normalization again; the decoder unit repeatedly calculates for 7 times, and a data feature vector is output by the last layer of the decoder unit; the score obtained by the Softmax layer is converted into a probability of a future water quality data trend, and the highest probability value is selected as a final output water quality prediction value; Step 5, inputting the training set data and iteratively updating and training according to a loss function loss until the preset iteration number of the model, terminating the training after the last iteration training; and screening a model with minimum absolute error and mean standard error in the training model to save the model structure; inputting the test set data into the saved model structure to verify the effect of the saved model.

2. The water quality prediction method based on transformer sparse attention mechanism according to claim 1, characterized in that: When the water quality data and information are obtained, a front-end PLC collection device is arranged at a preset collection site to sample in a specified river, lake or other water area to obtain water quality data; and an original database is constructed according to the water quality data set.

3. The water quality prediction method based on transformer sparse attention mechanism according to claim 1 or 2, characterized in that: The water quality data includes COD, BOD, total phosphorus content, total nitrogen content and ammonia nitrogen content information.

4. The water quality prediction method based on transformer sparse attention mechanism according to claim 1, characterized in that: The missing data is filled in by the multiple interpolation method, and then the complete data set is subjected to mean value normalization processing, and the mean value normalization formula is: ； wherein denotes the average value of all numbers, value denotes the value taken at the current time instant. 5.The water quality prediction method based on transformer sparse attention mechanism according to claim 1, characterized in that: Periodic characteristics of data are extracted by a sliding average filtering method, wherein the extraction method is: Obtaining trend information from input data of length d , The periodic decomposition formula is: ； final processed time data sequence ; wherein, is the number of occurrences of season i, is is the total number of occurrences of season i and trend j, is the trend term, is the noise, is the value of the time series data information taken, is the average of the sum of all Ds.

6. The water quality prediction method based on the transformer sparse attention mechanism according to claim 1, characterized in that: The matrix vector is input into the Encoder encoder part of the constructed model, the score is extracted through the sparse attention feature, the multi-segment local data is obtained in the global data, and the probability distribution of each local data is determined according to the autocorrelation degree of the global data; the score formula of the attention is: ； where d is the vector dimension, i.e., the number of columns of the matrix, Q represents the query vector, K represents the key vector, and V represents the value vector, denotes a sparse matrix resulting from the sparsity measure.

7. The method of claim 1, wherein the transformer-based sparse attention mechanism is a Transformer-based sparse attention mechanism. The Decoder decoder comprises two sparse attention modules, one feedforward neural network layer and three residual connection and normalization layers, and a single sparse attention mechanism module comprises a plurality of sparse attention mechanisms. 8.The water quality prediction method based on transformer sparse attention mechanism according to claim 1, characterized in that: In step 5, the predicted value and the true value are iteratively updated and trained according to the loss function loss, and the mean absolute error and the mean standard error are taken as the evaluation values; When the number of iterations reaches the preset number of iterations of the constructed transformer sparse attention mechanism model, the training is stopped; the time series prediction model with the minimum absolute error and mean standard error in the training model is screened out, and the model structure is saved.

Images