Water quality parameter measurement method driven by multilayer deep learning model

Abstract

The invention discloses a water quality parameter measurement method driven by a multilayer deep learning model, which realizes high-precision measurement of key water quality parameters such as COD (Chemical Oxygen Demand), total nitrogen (TN), total phosphorus (TP), nitrate nitrogen (NO3-N), nitrite nitrogen (NO3-N), ammonia nitrogen (NH3-N), turbidity and chromaticity by fusing CNN (convolutional kernel 3 * 1 and kernel number 32 / 64), BiLSTM (hidden nodes 30-60) and Attention mechanism and combining WOA hyper-parameter optimization (learning rate 0.001-0.01). According to the method, Savitzky-Golay filtering (window 11, order 3) denoising is adopted, data quality is enhanced through normalization preprocessing, model robustness is improved through 10-fold cross validation and an Adam optimizer, the problem that a traditional model is poor in adaptability to complex spectrum time sequence data is solved, and the method is suitable for rapid online monitoring of water quality.

Description

Technical Field

[0001] The present invention relates to the field of water quality monitoring technology, and in particular to a water quality parameter measurement method based on a multi-layer deep learning model, which is particularly suitable for measuring chemical oxygen demand (COD), total nitrogen (TN), total phosphorus (TP), nitrate nitrogen (NO3 - -N), nitrite nitrogen (NO2 - High-precision prediction of key water quality parameters such as nitrogen (NH3-N), ammonia nitrogen (NH3-N), turbidity and colority. Background Art

[0002] Water pollution has become a core challenge in global environmental governance, directly affecting ecological security and human health. Accurate monitoring of water quality parameters is the basis for pollution assessment, early warning and governance. Chemical oxygen demand (COD) is the core indicator for measuring organic pollution, reflecting the total amount of organic matter in water bodies. Too high a level of COD will lead to a decrease in the self-purification capacity of water bodies and even black and smelly phenomena. Total nitrogen (TN) and total phosphorus (TP) are the main causes of eutrophication. Excessive nitrogen and phosphorus will trigger algae outbreaks, leading to hypoxia in water bodies and loss of biodiversity. Nitrate nitrogen (NO3 - -N) and nitrite nitrogen (NO2 - -N) mainly comes from agricultural runoff and industrial wastewater, and long-term intake may cause methemoglobinemia and carcinogenic risks; ammonia nitrogen (NH3-N) is highly toxic to aquatic organisms and is easily converted into nitrite, exacerbating the toxicity of water bodies. In addition, turbidity characterizes the content of suspended particles, which directly affects light penetration and sensor measurement accuracy; colorimetry reflects the concentration of dissolved organic matter and metal ions, both of which jointly affect the sensory quality and ecological function of water bodies. However, water quality testing faces multiple challenges: parameter diversity complicates the detection scheme (such as differences in the sources of COD and TN), the coexistence of multiple parameters causes spectral overlap and interference (such as the superposition of humic acid and COD absorption peaks, and turbidity scattering masks the true absorbance), and dynamic variability (such as seasonal flow fluctuations) requires the model to have nonlinear time series adaptability.

[0003] Traditional detection methods have significant limitations in addressing the above challenges. Chemical analysis methods such as the potassium permanganate titration method for COD are cumbersome and require highly corrosive reagents, posing a risk of secondary contamination; spectrophotometric methods (such as alkaline potassium persulfate digestion-UV spectrometry for TN) have complex steps and their sensitivity is easily affected by reagent purity and operational errors; electrode methods require frequent calibration due to ion interference and short lifespan, making it difficult to meet real-time monitoring needs. Among physical instrument methods, although UV-visible spectroscopy can quickly detect COD, the scattering effect and chromatic interference caused by turbidity lead to baseline drift, reducing measurement accuracy; near-infrared spectroscopy is not sensitive enough to trace pollutants and has poor model generalization capabilities. Sensor technologies (such as optical and electrochemical sensors) are easily affected by biological attachment and temperature fluctuations and lack long-term stability. Existing methods mostly focus on a single parameter and lack comprehensive consideration of the coupling relationship between multiple parameters. For example, the positive correlation between COD and TN has not been effectively utilized, resulting in low detection efficiency.

[0004] In recent years, machine learning and deep learning technologies have been gradually applied to water quality prediction, but they still face bottlenecks. Traditional models (such as support vector regression and random forests) rely on manual feature engineering and have difficulty capturing the joint spatial-temporal features in spectra. Unidirectional long short-term memory networks (LSTMs) or convolutional neural networks (CNNs) can only process single-dimensional information and are unable to collaboratively extract the spatial distribution (such as absorption peak positions) and temporal evolution (such as pollutant concentration fluctuations) of spectra. In addition, the optimization of model hyperparameters (such as the number of convolution kernels and learning rate) relies on grid search or genetic algorithms, which are computationally expensive and prone to falling into local optimality. Existing research often focuses on a single parameter (such as COD or ammonia nitrogen), ignoring the interactions between multiple parameters, resulting in limited practicality. More seriously, interference factors such as turbidity and color are not compensated for in the model, significantly increasing prediction errors in complex water samples. Current technological trends indicate that multi-source data fusion and adaptive optimization are key to improving monitoring efficiency, but issues such as collaborative extraction of multidimensional features, multi-task joint learning, and lightweight deployment remain unresolved. For example, there is an urgent need to develop composite models that can simultaneously capture spectral spatial characteristics, temporal variations, and environmental interference. By sharing underlying features, efficient parallel prediction of parameters such as COD, TN, and TP can be achieved, avoiding the waste of duplicate modeling resources. At the same time, existing deep learning models have a large number of parameters, making them difficult to deploy on edge devices (such as portable spectrometers). Therefore, a combination of model compression and transfer learning techniques is needed to adapt to real-time monitoring needs. Summary of the Invention

[0005] To address the above problems, the present invention proposes a water quality parameter measurement method driven by a multi-layer deep learning model. By integrating CNN, BiLSTM and attention mechanism (Attention), combined with the whale optimization algorithm (WOA), it breaks through the technical bottlenecks of multi-parameter joint prediction, anti-interference design and dynamic hyperparameter optimization. The model introduces a turbidity-chromaticity compensation module to correct physical interference, uses WOA to adaptively adjust parameters such as the number of convolution kernels and learning rate, and compresses the model size through parameter pruning and quantization technology, ultimately achieving chemical oxygen demand (COD), total nitrogen (TN), total phosphorus (TP), nitrate nitrogen (NO3 - -N), nitrite nitrogen (NO2 - The high-precision simultaneous prediction and lightweight deployment of key water quality indicators such as nitrogen (NH3-N), ammonia nitrogen (NH3-N), turbidity and color provide innovative solutions for real-time monitoring of complex water quality.

[0006] The present invention is achieved through the following technical solutions:

[0007] During data preprocessing, the raw spectra were smoothed using a Savitzky-Golay filter (window length 11, polynomial order 3) to address noise and physical distortion in the UV-visible spectrum. Spectral intensities were mapped to the [0, 1] interval through maximum-minimum normalization to eliminate the interference of dimensional differences on model training. Furthermore, a linear compensation matrix was constructed based on the measured turbidity and color to dynamically correct the absorbance data, reducing the impact of turbidity scattering and color superposition on spectral authenticity and improving data quality for complex water samples.

[0008] In model construction, a multi-layer deep learning architecture was designed that integrates a convolutional neural network (CNN), a bidirectional long short-term memory network (BiLSTM), and an attention mechanism. The CNN module comprises two layers of convolution operations (with 32 and 64 kernels of 3×1 size, respectively). It extracts spatial distribution features of spectral data (such as absorption peak positions and intensity variations) through local connections and weight sharing. The BiLSTM module processes time series data bidirectionally using forward and backward long short-term memory units (48 hidden layer nodes), capturing the dynamic evolution of pollutant concentrations (such as seasonal fluctuations and sudden pollution events). The Attention module generates attention weights by performing a dot product operation between a trainable query vector and the BiLSTM output. Softmax normalization is used to dynamically weight key wavelength regions (such as COD-sensitive bands) to enhance the model's ability to autonomously discriminate feature importance. Finally, a fully connected layer maps the weighted context vector to a multi-parameter output space, enabling simultaneous prediction of eight water quality indicators, including COD, TN, and TP.

[0009] During model optimization and training, the Whale Optimization Algorithm (WOA) was introduced to achieve adaptive hyperparameter tuning. By setting the population size to 10 and the maximum number of iterations to 300, and using the validation set root mean square error (RMSE) as the optimization target, a global search was conducted for the number of convolution kernels (32-64), the number of BiLSTM hidden layer nodes (30-60), the learning rate (0.001-0.01), and the regularization coefficient (0.0001-0.001), resulting in the optimal parameter combination (56 convolution kernels, 52 BiLSTM nodes, a learning rate of 0.0032, and a regularization coefficient of 0.0005). The model was trained using the Adam optimizer with an initial learning rate of 0.0032 and a per-cycle decay coefficient of 0.95. The loss function was the multi-task weighted mean square error (COD weight 0.4, TN and TP weights 0.2 each, and other parameters 0.05). An early stopping strategy (terminating after 20 consecutive validation cycles without improvement in the loss) was introduced to prevent overfitting.

[0010] To further improve engineering practicality, the trained model was lightweighted. By removing weight parameters with absolute values ​​less than 0.001 in the fully connected layer through amplitude pruning, the model sparsity was increased to 70%; 8-bit integer quantization technology was used to compress 32-bit floating-point parameters to 35% of the original volume. The optimized model is suitable for embedded devices such as Raspberry Pi 4B, with an inference speed of 120 times / second, supporting real-time online monitoring. In transfer learning, for data from new water areas (such as the Taihu Lake Basin), the underlying CNN weights are frozen and the fully connected layer is fine-tuned (learning rate 0.0001, training cycle 50) to achieve efficient adaptation of water quality parameters across regions.

[0011] The method was experimentally verified, and the prediction coefficient of COD (R 2 ) reached 0.962, and the root mean square error (RMSE) was 0.131; the R 2 were 0.941 and 0.928 respectively, and the average R 2 For water samples with high turbidity (>50 NTU) and high chroma (>100 degrees), the anti-interference module keeps the increase in prediction error within 10%, significantly improving the accuracy compared to the uncompensated model (error increase of 30%-42%).

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] 1. High-precision prediction: The coefficient of determination (R 2 ) reached 0.9601, and the root mean square error (RMSE) was 0.1339, which was significantly better than the traditional support vector regression (SVR, R 2 =0.8324) and single module models (such as BiLSTM-Attention, R 2=0.7854), especially for complex water samples (containing high turbidity or multiple pollutants coexisting), the prediction stability is significantly improved.

[0014] 2. Efficient multi-task collaboration: By sharing underlying features, it achieves simultaneous prediction of parameters such as COD, TN, and TP, avoiding the resource consumption of repeated modeling. The detection efficiency is increased by more than 50% compared with traditional single-parameter methods, making it suitable for large-scale water quality monitoring scenarios.

[0015] 3. Strong anti-interference ability: The turbidity-color compensation module reduces the prediction error of the model by more than 30% compared with the uncompensated model in water samples with turbidity > 50NTU or chromaticity > 100 degrees, solving the pain point of existing spectroscopy methods that are susceptible to physical interference.

[0016] 4. Automation and low computing costs: WOA hyperparameter optimization reduces tuning time by 70%, and model training only requires conventional GPU resources (such as NVIDIA GTX 1080) without manual intervention, significantly reducing technical barriers and operation and maintenance costs.

[0017] 5. Lightweight and practical: The compressed model can run on embedded devices such as the Raspberry Pi, with an inference speed of 100 times per second, supporting real-time monitoring and providing a feasible solution for scenarios such as watershed inspections and online monitoring of sewage treatment plants.

[0018] 6. Environmental friendliness: Non-contact detection based entirely on UV-visible spectroscopy, without the need for chemical reagents, avoids the risk of secondary contamination of traditional titration or electrode methods, and is in line with the development trend of green detection technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 : Example of UV-Vis spectral data;

[0020] Figure 2 : Schematic diagram of the experimental setup;

[0021] Figure 3 : UV absorption spectra before and after SG filtering;

[0022] Figure 4 : CNN model structure diagram;

[0023] Figure 5 : Algorithm flow chart;

[0024] Figure 6 : Comparison between COD prediction values ​​and true values ​​in model training and test sets DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0026] Example 1: Model construction and training

[0027] Step 1: Data collection and preprocessing

[0028] 1. Water sample collection: Water samples were collected from typical surface areas, covering different seasons and pollution levels. A total of 500 groups of samples were obtained, and COD, TN, TP, and NO3 were measured simultaneously in each group of samples. - -N, NO2 - -N, NH4-N, turbidity and color parameters.

[0029] 2. Spectral Acquisition: A spectral measurement system was constructed using a deuterium halogen lamp or pulsed xenon lamp (wavelength range 185-2500 nm) and a spectrometer (wavelength range 200-1100 nm). The integration time was set to 100 ms, and 10 scans were averaged. Data was transmitted via an SMA905 optical fiber. Each water sample was measured five times, and the average value was used as the raw spectral data.

[0030] 3. Data preprocessing:

[0031] Denoising: Savitzky-Golay filtering (window length 11, polynomial order 3) was used to eliminate high-frequency noise and retain characteristic absorption peaks such as COD at 260 nm, TN at 220 nm, and TP at 880 nm.

[0032] Normalization: Normalize the spectral intensity to the maximum and minimum values ​​and map it to the [0,1] interval.

[0033] Interference compensation: Based on the turbidity and colorimetric data, a linear compensation matrix is ​​constructed to correct the spectral absorbance value.

[0034] Step 2: Model Architecture Design

[0035] 1. Input layer: Receives preprocessed spectral data (dimension 1×801, corresponding to wavelength 155-1000nm, resolution 1nm).

[0036] 2.CNN module:

[0037] First convolutional layer: 32 3×1 convolution kernels, stride 1×1, ReLU activation, output dimension 32×799.

[0038] Max pooling layer: pooling window 2×1, stride 2×1, output dimension 32×399.

[0039] Second convolutional layer: 64 3×1 convolution kernels, stride 1×1, ReLU activation, output dimension 64×397.

[0040] 3. BiLSTM module: Bidirectional LSTM layer: 48 hidden layer nodes (24 each for forward and backward), 1 time step, and output dimension 48×2 (bidirectional concatenation).

[0041] 4.Attention module:

[0042] Generate a trainable query vector (dimension 64), perform dot product operation with BiLSTM output, perform Softmax normalization and weighted summation, and output context vector (dimension 64).

[0043] 5. Fully connected layer and output layer:

[0044] Fully connected layer: 64 nodes, ReLU activation.

[0045] Output layer: 8 nodes (corresponding to 8 water quality parameters), linear activation.

[0046] Step 3: Hyperparameter Optimization

[0047] 1.WOA parameter settings:

[0048] Optimization goal: minimize the validation set RMSE (weighted average of COD, TN, TP and other parameters).

[0049] Population size: 10, maximum number of iterations: 300, variable dimension: 4 (number of convolution kernels 32-64, BiLSTM nodes 30-60, learning rate 0.001-0.01, regularization coefficient 0.0001-0.001).

[0050] 2. Optimization process:

[0051] Initialize the whale population position and calculate the fitness (RMSE of the validation set after model training).

[0052] Iterative update: adjust parameters according to the spiral bubble network strategy, retain the top 10% optimal solutions, and converge after 200 iterations.

[0053] The optimal parameter combination is: 56 convolution kernels, 52 BiLSTM nodes, 0.0032 learning rate, and 0.0005 regularization coefficient.

[0054] Step 4: Model training

[0055] 1. Data partitioning: Divide the data into training set (350 groups), validation set (100 groups), and test set (50 groups) in a ratio of 7:2:1.

[0056] 2. Training configuration:

[0057] Optimizer: Adam, initial learning rate 0.0032, decay coefficient 0.95 per cycle.

[0058] Loss function: multi-task mean square error (MSE), weighted coefficients (COD: 0.4, TN: 0.2, TP: 0.2, other parameters are 0.05 each).

[0059] Batch size: 32, maximum training epochs: 500, early stopping strategy (termination after 20 consecutive epochs with no improvement in validation loss).

[0060] 3. Performance evaluation:

[0061] Test set results: COD(R 2 =0.962, RMSE=0.131), TN(R 2 =0.941, RMSE=0.089), TP(R 2 =0.928, RMSE=0.076), and the average R 2 >0.90.

[0062] Example 2: Model Lightweighting and Real-time Deployment

[0063] Step 1: Model compression

[0064] 1. Parameter pruning: Prune the weights of the fully connected layer by removing weights with an absolute value less than 0.001, and increase the sparsity to 70%.

[0065] 2. Quantization: Convert 32-bit floating-point parameters to 8-bit integers (INT8), reducing the model size to 35% of its original size.

[0066] Step 2: Transfer Learning Adaptation

[0067] 1. New water area data adaptation:

[0068] 100 groups of surface water samples were collected, 10% of the original model weights (bottom layer CNN) were retained, the remaining layers were frozen and the fully connected layer was fine-tuned (learning rate 0.0001, 50 cycles).

[0069] 2. Performance verification:

[0070] Test set results: COD(R 2 =0.948, RMSE=0.152), TN(R 2 =0.927, RMSE=0.102), and the inference speed reached 120 times / second (NVIDIA Jetson Nano).

[0071] Step 3: Edge device deployment

[0072] 1. Hardware configuration:

[0073] Embedded device: Industrial computer equipped with a USB spectrometer (Ocean Insight FX series).

[0074] 2. Software deployment:

[0075] The model is converted to TensorFlow Lite format and integrated into the Python monitoring program, supporting real-time data collection, prediction, and result visualization.

[0076] 3. Field testing:

[0077] Continuous monitoring for 24 hours at the water inlet of a sewage treatment plant in Chongqing showed that the average deviation between the predicted results and the laboratory measured values ​​was less than 5%, meeting the requirements of the GB 3838-2002 surface water standard.

[0078] Example 3: Anti-interference performance verification

[0079] 1. Experimental Design

[0080] High turbidity water samples: Artificially prepare a turbidity gradient (0-100 NTU) and measure the COD prediction error.

[0081] High chromaticity water samples: Add humic acid to simulate chromaticity (0-150 degrees) and measure the TN prediction error.

[0082] 2. Result analysis:

[0083] Uncompensated model: When turbidity > 50 NTU, COD prediction error increases by 42%; when chromaticity > 100 degrees, TN error increases by 35%.

[0084] Compensated model: The error increase is controlled within 10%, proving that the turbidity-color module effectively suppresses physical interference.

[0085] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A water quality parameter measurement method driven by a multi-layer deep learning model, characterized in that: The following steps are involved: Step 1: Collect UV-visible absorption spectrum data of water samples and perform preprocessing, including: (1) The original spectral data were denoised using Savitzky-Golay filtering, with a filter window length of 11 and a polynomial order of 3; (2) Perform maximum-minimum normalization on the filtered spectral data and map the spectral intensity values ​​to the interval of 0, 10, and 1; Step 2: Build a multi-layer deep learning model, which includes: (1) CNN module: contains two convolutional layers. The number of convolution kernels in the first layer is 32, and the number in the second layer is 64. The convolution kernel size is 3×1, the stride is 1×1, and the activation function is ReLU. (2) BiLSTM module: contains forward and backward long short-term memory networks, with 30 to 60 hidden layer nodes and a time step of 1; (3) Attention module: Generates attention score by performing dot product operation between the trainable query vector and the BiLSTM output, and then performs weighted summation after Softmax normalization to output the context vector; Step 3: Use the Whale Optimization Algorithm (WOA) to optimize the hyperparameters of the model. The optimization parameters include: (1) Number of convolution kernels (32–64), number of BiLSTM hidden layer nodes (30–60), learning rate (0.001–0.01), and regularization coefficient (0.0001–0.001); (2) The WOA parameters are set as follows: population size 5, maximum number of iterations 200, and variable dimension 4; Step 4: Use the optimized model to perform feature extraction and time series analysis on the pre-processed spectral data, and output chemical oxygen demand (COD), total nitrogen (TN), total phosphorus (TP), nitrate nitrogen (NO3 - -N), nitrite nitrogen (NO3 - High-precision measurement of key water quality parameters such as nitrogen (NH3-N), turbidity and colority; Step 5: Train the model using 10-fold cross validation, gradient descent with the Adam optimizer, an initial learning rate of 0.001, a decay factor of 0.95 per training cycle, and a batch size of 32.

2. The water quality parameter measurement method driven by a multi-layer deep learning model according to claim 1 is characterized in that: The Savitzky-Golay filter has a window length of 11 and a polynomial order of 3, and is used to eliminate spectral noise and retain absorption peak characteristics.

3. The water quality parameter measurement method driven by a multi-layer deep learning model according to claim 1 is characterized in that: The convolutional layer of the CNN module is connected to a maximum pooling layer with a pooling window size of 2×1 and a stride of 2×1 to reduce the feature dimension and enhance translation invariance.

4. The water quality parameter measurement method driven by a multi-layer deep learning model according to claim 1 is characterized in that: The output dimension of the BiLSTM module is consistent with the query vector dimension of the Attention module. The query vector is generated by the fully connected layer and has a dimension of 64.

5. The water quality parameter measurement method driven by a multi-layer deep learning model according to claim 1 is characterized in that: In the WOA optimization process, the objective function is the root mean square error (RMSE) of the model on the validation set, and the spiral bubble net predation strategy is used to balance local search and global exploration.

6. The water quality parameter measurement method driven by a multi-layer deep learning model according to claim 1 is characterized in that: The evaluation indicators of the model include the coefficient of determination (R 2 ≥0.95), root mean square error (RMSE≤0.15), and mean absolute error (MAE≤0.12).

7. The water quality parameter measurement method driven by a multi-layer deep learning model according to claim 1 is characterized in that: The spectral data were collected at a wavelength of 155 nm to 1000 nm, with an integration time of 100 ms and an average number of scans of 10 times, and were transmitted to the spectrometer via an SMA905 interface optical fiber.

8. The water quality parameter measurement method driven by a multi-layer deep learning model according to claim 1 is characterized in that: The method is suitable for real-time water quality monitoring. It adapts to different water area data through transfer learning, deploys a lightweight model to compress the parameter scale to less than 60% of the original model, and realizes the chemical oxygen demand (COD), total nitrogen (TN), total phosphorus (TP), nitrate nitrogen (NO3 - -N), nitrite nitrogen (NO3 - High-precision measurement of key water quality parameters such as nitrogen (NH3-N), ammonia nitrogen (NH3-N), turbidity and colority.

Images