Water quality index prediction method based on multi-modal data fusion

Abstract

The application relates to a water quality index prediction method based on multi-modal data fusion, in particular to a water quality index prediction method based on a bottleneck attention module (BAM), low-rank multi-modal fusion (LMF), an attention mechanism, a convolutional neural network and an inverted transformer (ITransformer) model. First, image and water quality feature sequences are extracted from acquired remote sensing image data and water quality index historical data respectively. Then, the image and water quality features are fused based on the BAM and the LMF to obtain multi-modal fusion features. Finally, the multi-modal fusion features are input into the ITransformer model to predict future long-term water quality index values, and finally, a water quality index prediction result with high precision is obtained.

Description

TECHNICAL FIELD

[0001] The application relates to a water quality index prediction method, in particular to a water quality index prediction method based on multi-modal data fusion. BACKGROUND

[0002] With the progress of human society and the enhancement of public environmental protection awareness, scientific and reasonable use of water resources and comprehensive and systematic protection of water environment have become the inevitable choice for sustainable development of countries around the world. The water quality prediction method can obtain the short-term or long-term water quality and the future water quality change trend, provide guidance for water pollution prevention and control, provide health guidance for the public, and provide technical support for water environment governance. Water quality prediction is essentially a time series prediction problem, which refers to predicting the change of water quality index in the future according to the water quality index value at the historical time point. The research direction of water quality prediction can be divided into mechanism model and data-driven model. The mechanism model needs to pre-set a large number of parameters, and the training process is complex, which requires a large amount of computing resources and time cost. The data-driven model can be divided into statistical methods, machine learning methods and deep learning methods.

[0003] In recent years, deep learning methods have achieved good results in the field of time series prediction. Compared with machine learning which needs complex feature engineering, it can automatically learn the patterns and trends in time series data, and neural networks involve important parameters such as the number of hidden layers and the number of neurons, so deep learning models have very good ability to express complex nonlinear patterns. The Transformer regards the time steps of the input sequence as position information, represents the features of each time step as a vector, and uses an encoder-decoder framework for prediction. The frequency enhancement decomposition Transformer introduces a local attention mechanism and a reversible attention mechanism, so that the model can better capture the local features in the time series data and has higher computational efficiency. However, the actual influence of water quality index in water environment also includes meteorological, pollutants and other factors, and only considering the historical data of water quality itself is not comprehensive, so in order to make more accurate water quality prediction, other multi-modal data such as meteorological data also need to be considered, and the data information from different modalities needs to be fused. SUMMARY

[0004] In view of the above prior art deficiencies, the present application provides a water quality index prediction method based on a bottleneck attention module (BAM), low-rank multimodal fusion (LMF), attention mechanism, convolutional neural network, and inverted Transformer (ITransformer) model. The method includes a multimodal data fusion scheme based on BAM and LMF, and a water quality index multi-step prediction based on an ITransformer model. The purpose of the present application is achieved by the following technical solutions.

[0005] A water quality index prediction method based on multimodal data fusion, the method comprising the following steps:

[0006] 1) Obtain time series data of water quality indicators monitored in a river over a period of time and satellite remote sensing image data of rainfall in the river area;

[0007] 2) Extract image feature sequences and water quality feature sequences from the obtained remote sensing image data and water quality indicator historical data;

[0008] 3) Based on BAM and LMF, the image features and water quality features are fused based on the data in 2) to obtain multimodal fusion features;

[0009] 4) Based on the data in 3), the multimodal fusion features are input into the ITransformer model, and the multi-step prediction value of the water quality index is output, thereby obtaining the true future water quality index prediction value. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 A water quality index prediction method based on multimodal data fusion;

[0011] Figure 2 A water quality index prediction method based on multimodal data fusion; DETAILED DESCRIPTION

[0012] The features and exemplary embodiments of various aspects of the present application will be described in detail below. The following description covers many specific details to provide a thorough understanding of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced without some of the specific details. The following description of the embodiments is merely provided to provide a more clear understanding of the present application by showing examples of the present application. The present application is in no way limited to any specific configuration and algorithm presented below, but covers any modification, replacement and improvement of related elements, components and algorithms without departing from the spirit of the present application.

[0013] The specific steps of a water quality index prediction method based on multi-modal data fusion according to an embodiment of the present application will be described below with reference to the accompanying drawings. Figure 1

[0014] Firstly, time series data of water quality indexes monitored in a river in the past period of time and satellite remote sensing image data of rainfall in the river area are obtained.

[0015] Since the actual monitoring frequency of the water quality automatic monitoring system is usually once every 4 hours, the water quality parameter data is filtered and uniformly adjusted to 4-hour interval data in the data preprocessing stage. At the same time, the satellite remote sensing image data is uniformly adjusted to 4-hour interval data, representing the total rainfall in the past 4 hours.

[0016] For missing water quality index data, the data is filled by interpolation method.

[0017] Secondly, image and water quality feature sequences are extracted from the obtained remote sensing image data and water quality index historical data.

[0018] The water quality time series data is encoded by using an embedded structure, so as to obtain the water quality feature sequence. The remote sensing image data is feature-extracted by using a convolutional neural network, so as to obtain the image feature sequence.

[0019] Thirdly, based on BAM and LMF, the image and water quality features are fused to obtain multi-modal fusion features.

[0020] BAM is essentially a Transformer applied to multi-modal cases. A plurality of new tokens are introduced as "attention bottlenecks" in the input data. The input sequence becomes y=[y t ||y f ||y r ]. Different modalities can only share information and interact with each other through these bottleneck tokens, in which case y t and y r can only exchange information through y f . In order to reduce computational complexity, the model requires that the information flow of each modality be organized and compressed before passing through the bottleneck token, and share necessary information to ignore redundant information. The number of attention bottleneck tokens should be limited to much less than the tokens of the original input data. The bottleneck tokens are updated according to different modes, once for time series and once for remote sensing image, and finally the bottleneck tokens of each mode are averaged to obtain the final fusion tokens. This process can be defined as:

[0021]

[0022] ​wherein, Transformer(·) represents processing the input sequence using the Transformer model, and t , θ r represent the parameters of the model, and || represents connecting the input sequences together to form a new vector, represents the remote sensing features and time sequence features of the lth layer, represents the fusion bottleneck features of the lth layer, represents the remote sensing features and time sequence features of the l+1th layer obtained, represents the remote sensing and time sequence bottleneck features, represents the final obtained fusion bottleneck features.

[0023] In terms of fusion position, a mid-fusion strategy is adopted, assuming that the fusion layer is the nth layer, each mode of the first n-1 layers learns its own features using self-attention, and the calculation formula is:

[0024]

[0025] wherein, Transformer(·) represents processing the input sequence using the Transformer model, and t , θ r represent the parameters of the model, and || represents connecting the input sequences together to form a new vector, represents the remote sensing features and time sequence features of the lth layer, represents the remote sensing features and time sequence features of the k+1th layer obtained.

[0026] On the basis of self-learning, when reaching the fusion layer, mutual influence is achieved again through attention bottleneck. The calculation formula is as follows:

[0027]

[0028] y l+1 =Multimodal-Transformer(y l ;θ t , θ r )

[0029] wherein, Multimodal-Transformer(·) represents the fusion process mentioned in the above BAM.

[0030] In order to make the time series and remote sensing image achieve better fusion effect, the BAM is improved. The traditional BAM directly sums the multi-modal labels after interactive learning, and then averages the fusion result. However, this method has too many parameters, high computational complexity, and high risk of overfitting. In order to solve this problem, the low-rank fusion decomposes the weight tensor W into l groups of specific modal factors The dimension of each group of factors is T. In the case of effective decomposition, the minimum value of R is the rank of the tensor. This process can be defined as:

[0031]

[0032] Wherein, is the kth group of modal factors, M is the number of groups of modal factors, denotes the tensor product, R denotes the number of weight matrices participating in fusion, denotes the weight of the ith weight matrix on the mode m and the mode k.

[0033] In fact, the tensor Z also needs to be decomposed into in parallel with the modal specific factor, and the output tensor T can be obtained after operation without representing the tensor Z, thereby reducing the computational complexity of fusion.

[0034] Fourthly, the multi-modal fusion features are input into the ITransformer model, and the multi-step prediction value of the water quality index is output, so that the real future water quality index prediction value is obtained.

[0035] The ITransformer model is used to analyze the water environment related index, and after the data is processed in the above step, the input sequence The encoder is used to process the time series data of any length, extracts features from it, and predicts future time series data.

[0036] The ITransformer model is used to realize multi-index water quality prediction. Its structure is basically the same as that of the encoder of the traditional transformer. Due to the difference in embedding method and the change of the operation number from time token to index token, the functions of each module in the model have changed. In the transformer, the feedforward neural network learns the nonlinear characteristics of each indicator itself, thereby encoding the single indicator of its action, and decoding the future representation. The normalization layer is used for normalization processing of the index token, so that different index variables are kept in the same interval, thereby reducing the difference in numerical characteristics between different indexes.

[0037] In the traditional transformer, the attention mechanism calculates the attention of different positions of the input sequence to learn the context relationship and dependency in the sequence. In the transformer, the attention mechanism is used to capture the correlation between different indicator variables. The attention mechanism module linearly projects the indicator variables to obtain the query (q), key (k) and value (v) of the indicator label. Since the feature dimension of the indicator variable has been normalized, the attention mechanism calculates the correlation between variables by the following formula:

[0038]

[0039] Wherein, Softmax represents the Softmax function, d n represents the dimension of the vector, q represents the query vector, k represents the key vector, and v represents the value vector.

[0040] The water quality prediction model can be applied to the prediction of pH value, dissolved oxygen (DO), ammonia nitrogen (NH3-N), permanganate index (CODMN) and other water quality indicators of different rivers of surface water, realizes accurate prediction of related water quality data, and is convenient for water quality early warning and water pollution control.

[0041] The above detailed description of the technical solutions of the present application by means of the preferred embodiments should be understood as illustrative rather than limiting. Based on the description of the present application, those skilled in the art can modify the technical solutions recorded in each embodiment, or make equivalent replacement to part of the technical features, however, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present application.

Claims

1. A method for predicting water quality indicators based on multimodal data fusion, characterized in that, The method includes the following steps: 1) Obtain time-series data of water quality indicators monitored over a period of time for a river and satellite remote sensing image data of rainfall in the river area; 2) Extract image and water quality feature sequences from the acquired remote sensing image data and historical water quality index data; 3) Based on the bottleneck attention module (BAM) and the low-rank multimodal fusion method (LMF), image and water quality features are fused to obtain multimodal fusion features; 4) Input the multimodal fusion features into the inverted Transformer (i.e., ITransformer) model, and output the multi-step predicted values ​​of water quality indicators to obtain the predicted values ​​of future water quality indicators.

2. The method according to claim 1, characterized in that, A water quality prediction model is trained based on historical water quality time series data and historical rainfall remote sensing image data, including: Acquire time-series water quality data and remote sensing image data of rainfall in the target area as historical data; divide the historical data into training set and test set according to a preset ratio, and train the water quality prediction model based on the historical data of the training set to obtain the parameters of the water quality prediction model.

3. The method according to claim 1, characterized in that, Water quality prediction based on the aforementioned water quality prediction model includes: Acquire water quality time series data and rainfall satellite remote sensing image data for the target area at the current time and within a preset time period prior to the current time; input the data into the water quality prediction model; obtain water quality prediction data for the target area.

4. The method according to any one of claims 2 and 3, characterized in that, Water quality prediction models are constructed based on BAM, LMF, convolutional neural networks, and ITransformer models, including: Image data is processed using a convolutional neural network to obtain an image feature sequence; water quality data is processed using an embedding operation to convert the water quality data into a multi-dimensional continuous vector to obtain a water quality feature sequence; the above features are fused using BAM and LMF to obtain multimodal fusion features; the multimodal fusion features are used as input to the ITransformer model to construct the water quality prediction model.

5. The method according to claim 2, characterized in that, The water quality prediction model is tested and optimized based on historical data from the test set, including: The water quality prediction model is optimized by adjusting the number of attention points in the model and testing the adjusted model based on the test results.

6. The method according to claim 3, characterized in that, The feature sequence is divided into subsequences according to a preset sliding window width, including: The length of each subsequence is the width of the sliding window, which is the sum of the input time series length and the prediction time series length; The data captured by the sliding window is separated into input and predicted values. The sum of the input and predicted time series lengths is set manually, and the sequence is then transformed into supervised data.

7. The method according to claim 4, characterized in that, Before predicting water quality based on a water quality prediction model, the following steps are also included: The preset ratio is changed, and the historical data is re-divided into training set and test set according to the changed preset ratio; the water quality prediction model is trained based on the historical data of the re-divided training set, and the water quality prediction model is fine-tuned.

8. The method according to claim 4, characterized in that, Water quality prediction based on multimodal data fusion models includes: The lengths of the input and output sequences of the prediction model are not equal; adjusting the length of the input sequence can further optimize the accuracy of the model's predictions.

Images