Water quality COD (Chemical Oxygen Demand) detection method and detection system

Abstract

The invention discloses a water quality COD (Chemical Oxygen Demand) detection method and detection system based on an ultraviolet / fluorescence characteristic level fusion spectrum combined with a CARS-BO-LSSVM model. Through SG filtering (window 11 and order 3) and Z-score standardized preprocessing, 13 ultraviolet and 26 fluorescence characteristic wavelengths are screened by adopting a CARS algorithm, and are spliced into a 39-dimensional fusion matrix according to rows, and finally, a prediction model is constructed through Bayesian optimization LSSVM (gamma = 12.7, sigma = 0.045). The test set RMSE is 0.2578 mg.L <-1 >, the precision is improved by 16.5% compared with that of a single spectrum method, and the method is suitable for efficient and accurate detection of water quality COD.

Description

Technical Field

[0001] The present invention belongs to the technical field of water quality detection, and specifically relates to a water quality COD detection method and system based on ultraviolet / fluorescence fusion spectroscopy combined with a CARS-BO-LSSVM model, which is suitable for rapid and high-precision detection of organic pollutants in water bodies. Background Art

[0002] 1. Traditional COD detection methods and their limitations

[0003] Chemical Oxygen Demand (COD) is a core indicator for measuring the total amount of organic pollutants in water bodies. Its detection is crucial to environmental protection, industrial wastewater management and drinking water safety. At present, the internationally accepted standard detection method is the wet chemical method (such as potassium permanganate titration and potassium dichromate oxidation method). This type of method uses a strong oxidant to oxidize organic matter in water samples under high temperature and acidic conditions, and then determines the amount of oxidant consumed by titration or colorimetry to calculate the COD value. Although the wet chemical method has high accuracy and reproducibility, it has significant defects:

[0004] Seriously time-consuming: The oxidation reaction takes several to dozens of hours, which cannot meet the needs of real-time monitoring;

[0005] Secondary pollution: The use of strong acids and heavy metal catalysts (such as silver sulfate) will produce toxic waste liquid;

[0006] Complex operation: It requires professional experimental personnel to operate and is sensitive to interference factors such as turbidity and color of water samples.

[0007] Therefore, the development of fast, environmentally friendly and automated COD detection technology has become an urgent need in the industry.

[0008] 2. Application and Challenges of Spectroscopy in COD Detection

[0009] The spectral method establishes a quantitative relationship between the spectral signal and the COD value by analyzing the absorption or emission characteristics of water samples at specific wavelengths of light. It has the advantages of being non-destructive, reagent-free, and capable of online monitoring. The main technologies currently include:

[0010] Ultraviolet absorption spectroscopy: Based on the absorption characteristics of the conjugated structure of organic matter in the ultraviolet band (200-400nm), it directly correlates absorbance with COD concentration. However, it is easily interfered by turbidity and suspended matter scattering, and has low sensitivity to non-chromogenic organic matter (such as sugars).

[0011] Fluorescence spectroscopy: COD is characterized by the fluorescence signal generated by aromatic organic compounds under laser excitation (e.g., excitation wavelength 405nm, emission wavelength 420-600nm). However, fluorescence intensity is significantly affected by temperature, pH, and quenching effects, and it is difficult to distinguish multiple fluorescent substances in complex water samples.

[0012] Near-infrared spectroscopy: Indirect analysis through the vibrational absorption of CH and OH bonds in organic matter, but the signal is weak and requires the support of complex chemometric models.

[0013] Although the single spectral method partially solves the defects of the traditional method, it still has the following bottlenecks:

[0014] Weak anti-interference ability: Environmental factors (such as turbidity, temperature) and the complexity of water sample components cause spectral signal fluctuations;

[0015] Nonlinear relationship: There is a nonlinear mapping between spectral characteristics and COD concentration, and the traditional linear model (such as PLS) has insufficient fitting ability;

[0016] Low information utilization: A single spectrum only reflects some of the characteristics of organic matter, making it difficult to fully characterize the degree of pollution in complex water samples.

[0017] 3. Current status and problems of multispectral fusion technology

[0018] To overcome the limitations of single spectrum methods, multi-spectral fusion technology has become a research hotspot. Fusion strategies are usually divided into three levels:

[0019] Data-level fusion: Directly concatenates raw spectral data, for example, merging UV and fluorescence spectra into a high-dimensional matrix. However, the dimensional differences between spectra can cause large variables to obscure key information, and data redundancy is high, making the model prone to overfitting.

[0020] Feature-level fusion: Extracting the characteristic wavelengths of each spectrum and then fusing them, for example, through principal component analysis (PCA) dimensionality reduction. This method can reduce redundant information, but traditional feature extraction algorithms (such as PCA and SPA) rely on linear assumptions and have difficulty capturing nonlinear features.

[0021] Decision-level fusion: Build prediction models for each spectrum separately, then combine the results through weighted averaging or voting. This method is computationally complex and requires a large number of models, limiting its practicality.

[0022] Existing research shows that feature-level fusion has potential in balancing information utilization and computational efficiency, but key issues remain unresolved:

[0023] Insufficient feature selection algorithms: Traditional methods (such as PCA) cannot effectively screen characteristic wavelengths that are strongly correlated with COD;

[0024] Single fusion strategy: Simply concatenating features may lead to the curse of dimensionality and lack normalization.

[0025] Poor model generalization: The fused high-dimensional data requires complex nonlinear model fitting, but parameter optimization relies on experience and is prone to falling into local optimality.

[0026] 4. Progress and bottlenecks of machine learning algorithms in spectral analysis

[0027] In recent years, machine learning algorithms (such as support vector machines and random forests) have been widely used in spectral data analysis, with their nonlinear modeling capabilities significantly improving prediction accuracy. Among them, the least squares support vector machine (LSSVM) has attracted considerable attention due to its high computational efficiency and robustness against overfitting. However, the performance of LSSVM is highly dependent on the selection of hyperparameters (such as the kernel function and regularization parameter), making traditional grid search or genetic algorithms (GAs) time-consuming and prone to missing the global optimal solution.

[0028] Bayesian optimization (BO), as a global optimization framework, has demonstrated advantages in hyperparameter tuning by constructing a probabilistic surrogate model of the objective function to efficiently explore the parameter space. However, existing studies have mostly applied BO to single spectral models and have not deeply integrated it with multispectral fusion techniques. Furthermore, while spectral feature selection algorithms (such as CARS) can screen for key wavelengths, their collaborative optimization mechanism with BO-LSSVM remains unclear, resulting in the model's potential not being fully explored. Summary of the Invention

[0029] To address the above technical bottlenecks, the present invention proposes a water quality COD detection method based on ultraviolet / fluorescence feature-level fusion combined with CARS-BO-LSSVM. This method significantly outperforms existing technologies in detection accuracy, anti-interference performance, computational efficiency, and environmental friendliness. It provides an efficient solution for rapid and accurate monitoring of water quality COD, and is particularly suitable for real-time online detection of complex water bodies, with important engineering application value. Its core innovations include:

[0030] Highly robust feature extraction: Using the Competitive Adaptive Reweighted Sampling (CARS) algorithm, the characteristic wavelengths of UV and fluorescence spectra are dynamically screened through Monte Carlo iteration and exponential decay functions, eliminating noise interference and retaining key information (such as 13 UV and 26 fluorescence wavelengths).

[0031] Adaptive feature fusion: The filtered features are spliced ​​into a fusion matrix by row, and the dimension differences are eliminated through maximum-minimum normalization to enhance data expression capabilities;

[0032] Intelligent modeling optimization: Bayesian optimization is used to automatically search for LSSVM hyperparameters (sigma = 0.045, gamma = 12.7), addressing the subjectivity and inefficiency of traditional parameter selection and improving model generalization.

[0033] Experiments show that the RMSE of the proposed method on the test set is 0.2578 mg·L -1 , compared with single UV spectroscopy (RMSE = 0.3089 mg·L -1 ) and fluorescence spectroscopy (RMSE = 0.3095 mg·L -1 ) accuracy is improved by 16.5%, and its anti-interference ability is significantly better than existing fusion methods (such as PCA-BO-LSSVM).

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. Detection accuracy is significantly improved

[0036] Technology Comparison: Traditional single spectroscopy methods (UV or fluorescence) have limited detection accuracy due to incomplete information coverage. For example, UV spectroscopy has low sensitivity to non-chromogenic organic compounds (such as sugars), while fluorescence spectroscopy is susceptible to interference from quenching effects.

[0037] The present invention improves upon this approach by combining UV-fluorescence feature-level fusion with the CARS algorithm to screen key wavelengths (13 for UV and 26 for fluorescence), fully extracting complementary information. Experimental data show that the RMSE (RMSE) for the test set is 0.2578 mg·L⁻¹, a 16.5% improvement over both UV spectroscopy alone (RMSE = 0.3089) and fluorescence spectroscopy (RMSE = 0.3095). It also outperforms traditional fusion methods (e.g., PCA-BO-LSSVM, RMSE = 0.2974).

[0038] 2. Enhanced anti-interference capability

[0039] Technology comparison: Existing spectroscopy methods are easily interfered by environmental factors such as turbidity, temperature, and pH, and data-level fusion is prone to information overlap due to dimensional differences.

[0040] The present invention improves:

[0041] Preprocessing optimization: SG filtering (window 11, polynomial order 3) is used to improve the signal-to-noise ratio and effectively suppress high-frequency noise;

[0042] Feature-level fusion strategy: Using CARS to screen key wavelengths and normalize them (maximum-minimum normalization) to eliminate dimensional differences and prevent spectral variables with large values ​​from dominating the model.

[0043] Scattering peak processing: The fluorescence spectrum was processed using the segmented cubic conformal interpolation method to remove the interference of the scattering peaks at 405 nm and 465 nm.

[0044] Experimental verification: In water samples containing suspended matter, the MRE (7.68%) of the present invention is reduced by about 23% compared with the single spectral method (MRE≈10%), indicating that the anti-interference ability is significantly improved.

[0045] 3. Model generalization and computational efficiency optimization

[0046] Technology comparison: Traditional LSSVM relies on manual parameter adjustment (such as grid search), which is time-consuming and prone to falling into local optimality; existing feature selection algorithms (such as PCA) are insufficient for fitting nonlinear relationships.

[0047] The present invention improves:

[0048] Bayesian Optimization (BO): Using a Gaussian Process Regression (GPR) surrogate model and combining it with an exploration-exploitation balance coefficient (kappa = 2.576), it quickly converges to the optimal hyperparameters (gamma = 12.7, sigma = 0.045) within 50 iterations, taking only 30% of the time required by traditional methods.

[0049] Advantages of the CARS algorithm: By dynamically screening characteristic wavelengths through Monte Carlo iteration and exponential decay function, the compression rates of ultraviolet and fluorescence spectra are as low as 1.8% and 2.1%, respectively, reducing redundant information while retaining key features.

[0050] Experimental verification: In the test of cross-water source (Yangtze River, Jialing River, etc.), model R 2 It is stable above 0.93, indicating strong generalization.

[0051] 4. Environmental protection and practicality are outstanding

[0052] Technology comparison: The wet chemical method requires the use of strong acid and heavy metal catalysts, which will cause secondary pollution; the existing spectral fusion system is highly complex and difficult to promote.

[0053] The present invention improves:

[0054] Reagent-free detection: completely dependent on optical sensing technology to avoid chemical contamination;

[0055] Lightweight system: Using self-built spectral equipment (deuterium halogen lamp / pulsed xenon lamp, laser, fiber optic spectrometer), the cost is 40% lower than commercial equipment;

[0056] Real-time monitoring capability: single detection time is less than 1 minute, supporting online continuous monitoring.

[0057] 5. Scalability and compatibility

[0058] Technology comparison: Existing methods have poor adaptability to specific water qualities (such as high turbidity sewage), and model expansion requires redesign.

[0059] The present invention improves:

[0060] Modular design: Feature extraction (CARS), fusion (normalization), and modeling (BO-LSSVM) modules can be optimized independently to facilitate adaptation to different sensors (such as near-infrared spectroscopy);

[0061] Multi-scenario verification: In waters with different pollution levels such as Panlong Creek and Bai Creek, the model RMSE fluctuation range is less than 5%, indicating wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 : Comparison of SG filtering denoising effects (original spectrum vs smoothed spectrum);

[0063] Figure 2 : Schematic diagram of the optimal model structure (feature processing and normalization process);

[0064] Figure 3 : UV absorption spectrum CARS variable selection results;

[0065] Figure 4 : Results of CARS variable selection for fluorescence emission spectra;

[0066] Figure 5 : Relative error of test sets of different models;

[0067] Figure 6 : UV-fluorescence feature-level fusion combined with CARS-BO-LSSVM model regression results;

[0068] Figure 7 : Error bar graph of UV-fluorescence feature-level fusion combined with CARS-BO-LSSVM model. DETAILED DESCRIPTION

[0069] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0070] 1. Sample Collection and Processing

[0071] (1) Water sample source and collection

[0072] A total of 48 original water samples were collected from typical water areas, 12 from each water source, covering three types of samples: clean water (COD < 2 mg·L-1), lightly polluted (2-4 mg·L-1) and heavily polluted (> 4 mg·L-1).

[0073] Use sterile glass bottles to collect water samples 20 cm below the water surface, store them away from light, and transport them to the laboratory at 4°C. Complete the test within 24 hours.

[0074] (2) Chemical method for determination of COD physicochemical value

[0075] According to "Water quality - Determination of chemical oxygen demand - Potassium permanganate method" (GB 11892-89), the specific steps are as follows:

[0076] Reagent preparation: Prepare 0.01 mol·L-1 potassium permanganate standard solution, sulfuric acid solution (1:3 volume ratio) and sodium oxalate standard solution.

[0077] Oxidation reaction: Take 100 mL of water sample and add it into a 250 mL conical flask, add 10 mL of sulfuric acid solution and 20.00 mL of potassium permanganate solution, and heat in a boiling water bath for 30 minutes.

[0078] Titration analysis: Add 10.00 mL of sodium oxalate standard solution while hot, back-titrate with potassium permanganate solution to the pink end point, and record the consumption.

[0079] Calculate COD value: Calculate COD concentration according to the formula.

[0080] 2. Absorption and fluorescence spectrum collection

[0081] (1) Experimental setup and parameter settings

[0082] UV absorption spectrum collection:

[0083] Light source: deuterium halogen lamp or pulsed xenon lamp;

[0084] Sample tank: 10×10×30mm quartz cuvette;

[0085] Spectrometer: fiber optic spectrometer, integration time 11ms, averaging times 11 times;

[0086] Reference correction: Deionized water was used as the reference to deduct the baseline interference.

[0087] Fluorescence emission spectrum collection:

[0088] Excitation light source: 405nm single wavelength semiconductor laser;

[0089] Signal collection: Fluorescence signals are collected in an orthogonal 90° direction to avoid interference from excitation light scattering;

[0090] Scattering peak processing: The piecewise cubic conformal interpolation method is used to remove the interference peaks at 405nm (Rayleigh scattering) and 465nm (Raman scattering).

[0091] (2) Spectral data acquisition process

[0092] After the water sample was left to stand for 1 hour, the supernatant was collected and tested by UV absorption and fluorescence spectroscopy in turn;

[0093] The wavelength range of ultraviolet absorption spectrum is 200~800nm, and the range of fluorescence emission spectrum is 400~800nm;

[0094] Each sample was measured three times, and the average value was taken as the final spectral data.

[0095] 3. Spectral Data Preprocessing

[0096] (1) SG filtering denoising

[0097] Filter parameters: window size 11, polynomial order 3;

[0098] Processing effect: High-frequency noise suppression rate> 90%, signal-to-noise ratio (SNR) increased from the original 10dB to more than 30dB.

[0099] (2) Standardization

[0100] Method: Z-score standardization, the formula is:

[0101] X norm =σX-μ

[0102] Where μ is the mean of the spectral data and σ is the standard deviation;

[0103] Purpose: To eliminate the dimensional differences between UV and fluorescence spectra and to prevent variables with larger values ​​from dominating the model during fusion.

[0104] 4. Feature extraction and fusion

[0105] (1) CARS characteristic wavelength screening

[0106] UV absorption spectrum screening:

[0107] Monte Carlo iterations were performed 50 times, with 80% of the wavelength variables randomly sampled each time;

[0108] Attenuation function: exponential decay (λ = 0.05), eliminating wavelengths with weight < 0.1;

[0109] Screening results: 13 characteristic wavelengths, compression rate 1.8%.

[0110] Fluorescence emission spectroscopy screening:

[0111] The number of iterations is 50, and the variable subset corresponding to the minimum RMSECV value;

[0112] Screening results: 26 characteristic wavelengths, compression rate 2.1%.

[0113] (2) Feature-level fusion

[0114] Methods: The UV feature matrix and the fluorescence feature matrix were concatenated row by row to generate a fusion matrix;

[0115] Normalization: Use maximum-minimum normalization, the formula is:

[0116] X norm =(XX min ) / (X max -X min )

[0117] Ensure that the fused data range is in the [0,1] interval.

[0118] 5. BO-LSSVM model establishment and verification

[0119] (1) Dataset division

[0120] Training set: 33 samples (randomly selected, accounting for 70%);

[0121] Test set: 15 samples (randomly selected, accounting for 30%).

[0122] (2) Model parameter optimization

[0123] Kernel function: radial basis function (RBF);

[0124] Hyperparameter search range: gamma∈[0.001,100], sigma∈[0.001,100];

[0125] Bayesian optimization settings:

[0126] Surrogate model: Gaussian process regression (GPR, Matern 5 / 2 kernel function);

[0127] Iteration number: 50 times, convergence condition: RMSE change rate < 1%;

[0128] Optimal parameters: gamma = 12.7, sigma = 0.045.

[0129] (3) Model performance verification

[0130] Training set results: R 2 =0.9371, RMSE=0.2726mg·L-1, MRE=9.99%;

[0131] Test set results: R 2 =0.9377, RMSE=0.2578mg·L-1, MRE=7.68%;

[0132] Comparative experiment: Compared with single UV spectroscopy (RMSE = 0.3089) and fluorescence spectroscopy (RMSE = 0.3095), the accuracy is improved by 16.5%.

[0133] 6. Practical Application Examples

[0134] Scenario: Real-time monitoring of a section of the Yangtze River;

[0135] Operation process:

[0136] After collecting the water sample, let it stand for 1 hour and take the supernatant;

[0137] Simultaneously collect UV absorption spectra (200-800 nm) and fluorescence emission spectra (400-800 nm);

[0138] Perform SG filtering, normalization and CARS feature extraction;

[0139] Input the fusion matrix to the pre-trained BO-LSSVM model and output the COD concentration value;

[0140] Results: Detection time is less than 1 minute, and the error compared with the chemical method is less than 8%.

[0141] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A water quality COD detection method, characterized in that, The following steps are involved: (1) Synchronously collect the UV absorption spectrum and fluorescence emission spectrum of water samples; (2) Preprocessing the ultraviolet absorption spectrum and the fluorescence emission spectrum, including the following operations: - Savitizky-Golay (SG) filtering is used for denoising and smoothing, with a window size of 11 and a polynomial order of 3; - Z-score normalization of spectral data to eliminate dimension differences; (3) The competitive adaptive reweighted sampling (CARS) method is used to screen the characteristic wavelengths of the preprocessed UV absorption spectrum and fluorescence emission spectrum, including: -Generate wavelength variable subsets through 50 Monte Carlo iterations; -Use exponential decay function to dynamically remove wavelength variables whose weight is lower than the threshold of 0.1; - Based on the principle of minimizing the mean square error of cross validation (RMSECV), the number of characteristic wavelengths of the UV spectrum and the number of characteristic wavelengths of the fluorescence spectrum were screened to be 13 and 26 respectively; (4) Feature-level fusion of ultraviolet characteristic spectrum information and fluorescence characteristic spectrum information, specifically including: - Concatenate the UV feature matrix (dimension 13×n) and the fluorescence feature matrix (dimension 26×m) row by row to generate a fused spectral data matrix (dimension 39×k), where k = max(n,m); -Normalize the fusion matrix to ensure data distribution consistency; (5) Based on the fused spectral data matrix, a Bayesian optimized least squares support vector machine (BO-LSSVM) prediction model is constructed, which specifically includes: -Set the LSSVM kernel function to radial basis function (RBF), and the search range of hyperparameters sigma and gamma is 0.001 to 100; -Iterate 50 times through the Bayesian optimization algorithm, and the objective function is the root mean square error of the test set (RMSE) is minimized, and the convergence condition is that the error change rate is less than 1%; -Use the training set data to fit the model, and verify the model prediction performance through the test set, and output the COD concentration value of the water sample.

2. The method according to claim 1, characterized in that In step (2), the window size of the SG filter is 11, the polynomial order is 3, and the spectral signal-to-noise ratio after smoothing is increased to more than 3 times of the original data.

3. The method according to claim 1, characterized in that In the step (3), the number of Monte Carlo iterations of the CARS method is 50, and the characteristic wavelength compression rates of the ultraviolet spectrum and the fluorescence spectrum are 1.8% and 2.1%, respectively.

4. The method according to claim 1, wherein In step (3), the CARS method dynamically adjusts the attenuation function parameters to eliminate wavelength variables with weights lower than 0.1, and retains the RMSECV value of the variable subset less than 0.3 mg·L-1.

5. The method according to claim 1, wherein In step (4), the normalization process of feature-level fusion adopts maximum-minimum normalization, and the formula is: X norm =(X-X min ) / (X max -X min ) Ensure that the value range of each dimension of the fusion matrix is ​​in the interval [0,1].

6. The method according to claim 1, characterized in that In step (5), the parameter search of the Bayesian optimization algorithm uses Gaussian process regression (GPR) as a proxy model, and the exploration-exploitation balance coefficient (kappa) is set to 2.

576.

7. The method according to claim 1, characterized in that In step (5), during the hyperparameter optimization process of the BO-LSSVM model, the final optimal values ​​of gamma and sigma are 12.7 and 0.045, respectively.

8. A water quality COD detection system, characterized in that: include: Spectral acquisition module, equipped with a deuterium halogen lamp or pulsed xenon lamp light source and a semiconductor laser (wavelength 405nm), for synchronously acquiring ultraviolet absorption spectra and fluorescence emission spectra; Preprocessing module, performing SG filtering (window 11, order 3) and Z-score normalization; The feature extraction module uses the CARS algorithm to iterate 50 times to screen 13 characteristic wavelengths of the UV spectrum and 26 characteristic wavelengths of the fluorescence spectrum; Feature fusion module, which splices the UV and fluorescence feature matrices row by row and performs maximum-minimum normalization; Modeling and prediction module, which builds a COD concentration prediction model based on the BO-LSSVM algorithm, with a hyperparameter search range of 0.001 to 100; Output module, displays and stores COD test results and model performance indicators (R 2 , RMSE, MRE).

9. The system according to claim 8, characterized in that In the spectrum acquisition module, the fluorescence emission spectrum is collected in an orthogonal 90° direction to collect signals, and the scattering peak interference at 405nm and 465nm is removed by a segmented cubic conformal interpolation method.

Images