Water quality index prediction method based on multi-modal data fusion
Through the multimodal data fusion of BAM and LMF and the ITransformer model, the problem that existing water quality prediction methods fail to fully consider multimodal data is solved, and more accurate and efficient water quality indicator prediction is achieved, which is suitable for a variety of water quality indicators. predict.
Patent Information
- Application Number
- CN202410897865.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-05
AI Technical Summary
The existing water quality prediction methods rely solely on historical water quality data and fail to fully consider meteorological and pollution factors, resulting in inaccurate predictions, and deep learning models have high computational complexity in multimodal data fusion.
A multimodal data fusion scheme based on BottleneckAttention Module (BAM) and Low-rank Multimodal Fusion (LMF) was used, and combined with the inverted Transformer (ITransformer) model, multi-step water quality index prediction was performed by extracting water quality and remote sensing image features.
More accurate prediction of water quality indicators is achieved, the calculation complexity is reduced, the calculation efficiency and prediction accuracy of the model are improved, and it is suitable for the prediction of water quality indicators of different rivers such as pH value, dissolved oxygen and ammonia nitrogen.
Smart Images

Figure CN118941900A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a water quality index prediction method, and in particular to a water quality index prediction method based on multimodal data fusion. Background Art
[0002] With the progress of human civilization and the enhancement of public environmental protection awareness, the scientific and rational use of water resources and the comprehensive and systematic protection of the water environment have become the inevitable choice for sustainable development in all countries in the world. Water quality prediction methods can obtain short-term or long-term water quality and future water quality change trends, provide guidance for water pollution prevention and control, provide health guidance for the public, and provide technical support for water environment management. Water quality prediction is essentially a time series prediction problem, which means predicting the changes in water quality indicators in the future based on the water quality index values at historical time points. The research directions of water quality prediction can be divided into mechanism models and data-driven models. Mechanism models require a large number of parameters to be preset in advance, the training process is complex, and requires a lot of computing resources and time costs. Data-driven models can be divided into statistical methods, machine learning methods, and deep learning methods.
[0003] In recent years, deep learning methods have achieved good results in the field of time series prediction. Compared with machine learning that requires complex feature engineering, it can automatically learn patterns and trends in time series data, and neural networks involve important parameters such as the number of hidden layers and the number of neurons, so deep learning models have very good ability to express complex nonlinear patterns. Transformer regards the time step of the input sequence as position information, represents the features of each time step as a vector, and uses an encoder-decoder framework for prediction. The frequency-enhanced decomposition Transformer introduces a local attention mechanism and a reversible attention mechanism, which enables the model to better capture local features in time series data and has higher computational efficiency. However, the actual impact of water quality indicators in the water environment also includes factors such as meteorology and pollutants. Considering only the historical data of water quality itself is not comprehensive. Therefore, in order to make more accurate water quality predictions, other multimodal data, such as meteorological data, need to be considered, and data information from different modalities should be integrated. Summary of the invention
[0004] In view of the deficiencies of the above prior art, the present invention provides a water quality index prediction method based on Bottleneck Attention Module (BAM), Low-rank Multimodal Fusion (LMF), attention mechanism, convolutional neural network, and inverted Transformer (ITransformer) model. It includes: a multimodal data fusion scheme based on BAM and LMF; and multi-step prediction of water quality indicators based on the ITransformer model. The purpose of the present invention is achieved through the following technical solutions.
[0005] A water quality index prediction method based on multimodal data fusion, the method comprises the following steps:
[0006] 1) Obtain time series data of water quality indicators of a river monitored over a period of time and satellite remote sensing image data of rainfall in the river area;
[0007] 2) Extracting image feature sequences and water quality feature sequences from the acquired remote sensing image data and historical data of water quality indicators;
[0008] 3) Based on 2), image features and water quality features are fused based on BAM and LMF to obtain multimodal fusion features;
[0009] 4) Based on 3), the multimodal fusion features are input into the ITransformer model to output the multi-step prediction values of the water quality indicators, thereby obtaining the true future prediction values of the water quality indicators. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 Schematic diagram of the composition of a water quality index prediction method based on multimodal data fusion;
[0011] Figure 2 Model structure diagram of a water quality index prediction method based on multimodal data fusion. DETAILED DESCRIPTION
[0012] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. The following description covers many specific details in order to provide a comprehensive understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a clearer understanding of the present invention by illustrating examples of the present invention. The present invention is by no means limited to any specific configuration and algorithm proposed below, but covers any modification, replacement and improvement of related elements, parts and algorithms without departing from the spirit of the present invention.
[0013] The following will refer to the attached Figure 1 The specific steps of describing a water quality index prediction method based on multimodal data fusion according to an embodiment of the present invention are as follows:
[0014] The first step is to obtain the time series data of water quality indicators monitored in a river over a period of time and the satellite remote sensing image data of rainfall in the river area.
[0015] Since the actual monitoring frequency of the water quality automatic monitoring system is usually once every 4 hours, in the data preprocessing stage, the water quality parameter data is screened and uniformly adjusted to data with equal intervals of 4 hours. At the same time, the satellite remote sensing image data is uniformly adjusted to data with equal intervals of 4 hours, representing the total rainfall in the past 4 hours.
[0016] For missing water quality index data, interpolation method is used to fill in the data.
[0017] The second step is to extract image and water quality feature sequences from the acquired remote sensing image data and historical data of water quality indicators.
[0018] The embedded structure is used to encode the water quality time series data to obtain the water quality feature sequence. The convolutional neural network is used to extract the features of the remote sensing image data to obtain the image feature sequence.
[0019] The third step is to fuse the image and water quality features based on BAM and LMF to obtain multimodal fusion features.
[0020] BAM is essentially a Transformer applied to multimodal situations. Introducing multiple new tokens As the “attention bottleneck” in the input data. The input sequence becomes y = [y t ||y f ||y r ]. Different modalities can only share information and interact with each other through these bottleneck tokens, in which case y t and r Only through y f Exchange information. In order to reduce computational complexity, the model requires that the information flow of each modality be organized and compressed before passing through the bottleneck token, and share necessary information to ignore redundant information. The number of attention bottleneck tokens should be limited to be much smaller than the tokens of the original input data. The bottleneck tokens are updated separately according to different modes, once according to the time series and remote sensing images, and finally the bottleneck tokens of each mode are averaged to obtain the final fusion token. The process can be defined as:
[0021]
[0022] Among them, Transformer(·) represents the use of the Transformer model to process the input sequence, θ t ,θ r represents the parameters of the model, || represents connecting the input sequences together to form a new vector, represents the remote sensing features and time series features of the lth layer, represents the fusion bottleneck feature of layer l, represents the remote sensing features and time series features of the l+1th layer, Represents the bottleneck characteristics of remote sensing and time series, Represents the final fusion bottleneck feature.
[0023] In terms of fusion position, a mid-term fusion strategy is adopted. Assuming that the number of fusion layers is the nth layer, each mode in the first n-1 layers uses self-attention to learn its own features. The calculation formula is:
[0024]
[0025] Among them, Transformer(·) represents the use of the Transformer model to process the input sequence, θ t ,θ r represents the parameters of the model, || represents connecting the input sequences together to form a new vector, represents the remote sensing features and time series features of the lth layer, Represents the remote sensing features and time series features obtained at the k+1th layer.
[0026] On the basis of self-learning, when reaching the fusion layer, they influence each other through the attention bottleneck. The calculation formula is as follows:
[0027]
[0028] y l+1 =Multimodal-Transformer(y l θ t ,θ r )
[0029] Among them, Multimodal-Transformer(·) represents the fusion process mentioned in the above BAM.
[0030] In order to achieve a better fusion effect between time series and remote sensing images, BAM is improved. The traditional BAM directly sums the multimodal labels after interactive learning, and then averages the fusion results. However, the low-rank fusion idea is adopted. After interactive learning, the fusion weight of each modality is fully considered, and the multimodal labels are low-rank fused. A common fusion method that considers weights is tensor fusion. However, this method has too many parameters, complex calculations, and a high risk of overfitting. To solve this problem, low-rank fusion decomposes the weight tensor W into l groups of specific modality factors. The dimension of each group of factors is T. In the case of effective decomposition, the minimum value of R is the rank of the tensor. This process can be defined as:
[0031]
[0032] in, is the kth group of modal factors, M is the number of groups of modal factors, represents the tensor product, R represents the number of weight matrices involved in the fusion, Denotes the weight of the i-th weight matrix on mode m and mode k.
[0033] In fact, the tensor Z needs to be decomposed into In parallel with the modality-specific factors, the output tensor T can be operated to obtain the tensor Z that does not need to be represented, thereby reducing the computational complexity of the fusion.
[0034] The fourth step is to input the multimodal fusion features into the ITransformer model and output the multi-step prediction values of the water quality indicators to obtain the true future prediction values of the water quality indicators.
[0035] The present invention uses the ITransformer model to analyze water environment related indicators. After the data is processed in the previous step, the input sequence is set The encoder processes input time series data of any length, extracts features from it, and predicts future time series data.
[0036] The ITransformer model is used to achieve multi-indicator water quality prediction. Its structure is basically the same as the encoder of the traditional transformer. Due to the different embedding methods and the change of operands from time tokens to indicator tokens, the functions of each module in the model have changed. In the transformer, the feedforward neural network learns the nonlinear characteristics of each indicator itself, thereby encoding a single indicator of its action and decoding the future representation. The normalization layer is used to normalize the indicator tokens so that different indicator variables are kept in the same interval, thereby reducing the differences in numerical characteristics between different indicators.
[0037] In traditional transformers, the attention mechanism performs attention calculations on different positions of the input sequence to learn the contextual relationships and dependencies in the sequence. In transformers, the attention mechanism is used to capture the correlation between different indicator variables. The attention mechanism module linearly projects the indicator variables to obtain the query (q), key (k), and value (v) of the indicator tags. Since the feature dimensions of the indicator variables have been normalized, the attention mechanism calculates the correlation between variables through the following formula:
[0038]
[0039] Among them, Softmax represents the Softmax function, d n represents the dimension of the vector, q represents the query vector, k represents the key vector, and v represents the value vector.
[0040] This water quality prediction model can be applied to the prediction of water quality indicators such as pH value, dissolved oxygen (DO), ammonia nitrogen (NH3-N), and permanganate index (CODMN) of different surface water rivers, to achieve accurate prediction of relevant water quality data, and facilitate water quality early warning and water pollution control.
[0041] The present invention provides a water quality index prediction method based on multimodal data fusion proposed above. It should be understood that the detailed description of the technical solution of the present invention with the help of the preferred embodiment is illustrative rather than restrictive. A person of ordinary skill in the art may modify the technical solutions described in each embodiment based on reading the specification of the present invention, or replace some of the technical features therein by equivalents, however, these modifications or replacements do not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solution of each embodiment of the present invention.
Claims
1. A water quality index prediction method based on multimodal data fusion, characterized in that: The method comprises the following steps: 1) Obtain time series data of water quality indicators of a river monitored over a period of time and satellite remote sensing image data of rainfall in the river area; 2) Extract image and water quality feature sequences from the acquired remote sensing image data and historical data of water quality indicators; 3) Based on the Bottleneck Attention Module (BAM) and Low-rank Multimodal Fusion (LMF), the image and water quality features are fused to obtain multimodal fusion features; 4) The multimodal fusion features are input into the inverted Transformer (ITransformer) model to output the multi-step prediction values of water quality indicators, thereby obtaining the future prediction values of water quality indicators.
2. The method according to claim 1, characterized in that: The water quality prediction model is trained based on the historical water quality time series data and the historical rainfall remote sensing image data, including: Acquire water quality time series data and rainfall remote sensing image data of the target area as historical data; divide the historical data into a training set and a test set according to a preset ratio, and train the water quality prediction model according to the historical data of the training set to obtain the parameters of the water quality prediction model.
3. The method according to claim 1, characterized in that: The predicting of water quality based on the water quality prediction model comprises: Acquire water quality time series data and rainfall satellite remote sensing image data of the target area at present and within a preset time period before the current moment; input the data into the water quality prediction model; and obtain water quality prediction data of the target area.
4. The method according to claims 2 and 3, characterized in that: The water quality prediction model is constructed based on BAM, LMF, convolutional neural network and ITransformer model, including: The image data is processed using a convolutional neural network to obtain an image feature sequence; the water quality data is processed using an embedding operation to convert the water quality data into a multi-dimensional continuous vector to obtain a water quality feature sequence; the above features are fused using BAM and LMF to obtain multimodal fusion features; the multimodal fusion features are used as input to the ITransformer model to form the water quality prediction model.
5. The method according to claim 2, characterized in that: The step of testing and optimizing the water quality prediction model according to the historical data of the test set includes: The water quality prediction model adjusts the number of attention heads in the water quality prediction model and tests the adjusted water quality prediction model according to the test results of the water quality prediction model to optimize the parameters of the water quality prediction model.
6. The method according to claim 3, characterized in that: The step of dividing the subsequences according to the preset sliding window width as feature sequences includes: The length of each subsequence is the width of the sliding window, which is the sum of the input time series length and the predicted time series length; The data intercepted by the sliding window is separated into input values and predicted values. The sum of the input and predicted time series lengths can be set manually, and the sequence is converted into supervised data.
7. The method according to claim 4, characterized in that: Before predicting water quality based on the water quality prediction model, the method further includes: The preset ratio is changed, and the historical data is re-divided into a training set and a test set according to the changed preset ratio; the water quality prediction model is trained according to the historical data of the re-divided training set, and the water quality prediction model is fine-tuned.
8. The method according to claim 4, characterized in that The water quality prediction model based on multimodal data fusion includes: The lengths of the input and output sequences of the prediction model may not be equal; the length of the input sequence can be adjusted to further optimize the accuracy of the model prediction.
Citation Information
Patent Citations
Water quality prediction method and system based on deep learning space-time model
CN115146829A
Water quality prediction method and system based on GAT-Transform time sequence model
CN117273203A
Cited By
Land-based recirculating aquaculture water quality intelligent regulation and control system and method based on multi-modal perception
CN120257019A
Land-based recirculating aquaculture water quality intelligent control system and method based on multimodal perception
CN120257019B
