Sewage organic matter concentration online rapid detection method based on deep learning

Abstract

The invention belongs to the technical field of sewage quality monitoring, and particularly relates to an online rapid detection method for sewage organic matter concentration based on deep learning. According to the method, the turbidity compensation model is established to eliminate suspended matter interference, so that the turbidity interference resistance is remarkably improved, and the test precision of a sample under a complex water quality condition is improved; the method creatively associates the integral value of the ultraviolet visible spectrum with the organic matter concentration, avoids a response blind area of a single wavelength to a complex matrix, establishes a staging verification model from simple to complex, and adopts univariate linear regression and an XGBoost model to model a multiband integral feature set and the organic matter concentration, so as to improve the verification accuracy of the organic matter concentration. And the spectrum information utilization rate and accuracy are obviously optimized. The method provided by the invention solves the industrial problems of weak anti-interference capability, serious lagging, high cost and the like of the traditional organic matter concentration detection method, provides reliable technical guarantee for the realization of fine control and energy conservation and emission reduction targets in the sewage treatment process, and has remarkable economic benefits and social ecological values.

Description

Technical Field

[0001] This invention belongs to the field of wastewater quality monitoring technology, specifically relating to a method for rapid online detection of organic matter concentration in wastewater based on deep learning. Background Technology

[0002] Accurate determination of organic matter concentration is of irreplaceable value for the control of urban wastewater treatment processes. The traditional potassium dichromate method is widely used due to its high accuracy, but it suffers from technical bottlenecks such as being time-consuming, using highly corrosive reagents, and posing a risk of secondary pollution. Especially in response to sudden water pollution incidents, the lag of traditional detection methods makes it difficult to meet the stringent timeliness requirements of environmental emergency monitoring. Therefore, there is an urgent need for a rapid, accurate, environmentally friendly, and non-destructive method for detecting organic matter concentration to achieve online quantitative analysis of wastewater.

[0003] Spectroscopic analysis technology, with its rapid response, simultaneous detection of multiple parameters, and environmental friendliness, is widely used in water quality monitoring. Based on molecular absorption spectroscopy, establishing a mapping model between spectral characteristics and organic matter concentration by analyzing the quantitative relationship between light intensity attenuation in specific wavelength bands has become a current research hotspot. However, the interference effect of complex water matrix on spectral detection cannot be ignored. Background noise and high turbidity in spectral data constitute multiple interference sources affecting detection accuracy. Therefore, developing advanced spectral preprocessing algorithms and intelligent analytical models has become a key path to overcome technical bottlenecks. Deep learning technology provides a new paradigm for spectral data mining, enabling more comprehensive utilization of effective information in spectral data and improving the reliability of water quality parameter prediction and analysis.

[0004] Among existing spectroscopic techniques for detecting organic matter concentration, Chinese invention patent CN117890312A proposes using UV-Vis spectroscopy and fluorescence spectroscopy, combined with the LS-SVR algorithm to construct a COD model, thereby utilizing the effective information carried by the two spectra and improving the model's generalization performance. Chinese invention patent CN117874442A uses a genetic algorithm to construct a detection model based on characteristic wavelengths and COD content, enabling rapid and relatively accurate COD content measurement in water samples. Chinese invention patent CN117033938A uses a least-squares support vector machine as the detection model to construct a COD detection method, which removes data noise from the spectral data and uses the SG method to smooth the curves. Chinese invention patent CN118425076A discloses a turbidity compensation method for optical water quality detection and COD detection, and establishes the relationship between turbidity and absorbance contribution values ​​and wavelength through various data fitting methods. However, for COD concentration prediction, it only uses 254nm as the key ultraviolet wavelength and the most representative wavelength for COD detection. Chinese invention patent CN112326565B obtains the turbidity influence factor through compressed sensing theory, solves the optimal solution to reconstruct the turbidity absorption spectrum influence matrix, and subtracts the turbidity absorption spectrum to finally obtain the COD absorption spectrum.

[0005] While the aforementioned existing technologies can quickly and effectively predict organic matter concentration, two problems still exist. Current methods mostly rely on absorbance at a single wavelength or extract absorbance at only a few characteristic wavelengths, and the absorbance values ​​at a few characteristic wavelengths cannot fully cover all the information about organic matter. In addition, most current methods ignore the influence of turbidity, and only a few methods involve turbidity compensation. However, spectral data are easily affected by turbidity and have poor adaptability to complex water quality. Current single correction methods are still difficult to cope with complex and fluctuating water quality conditions. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a rapid online detection method for organic matter concentration in wastewater based on deep learning. To address the problems of large prediction errors and narrow applicability in existing spectroscopic organic matter concentration detection methods due to turbidity interference, limited feature extraction, and poor model generalization ability, this invention provides a rapid, efficient, and accurate method for detecting organic matter concentration in urban wastewater. Through spectral preprocessing, integral feature extraction, and deep learning model optimization, the prediction accuracy and anti-interference ability are improved.

[0007] To achieve the above technical objectives, the technical solution adopted in the embodiments of the present invention is as follows: A rapid online detection method for organic matter concentration in wastewater based on deep learning includes the following steps: Step S1: Collect wastewater organic matter concentration data and ultraviolet-visible spectrum at different times to make the organic matter concentration data correspond to the ultraviolet-visible spectrum data, and divide the complete dataset into training set and test set; Step S2: Perform multivariate scattering correction (MSC) preprocessing on the UV-Vis spectral data described in step S1 to obtain turbidity-compensated UV-Vis spectral data. Step S3: Perform multi-band piecewise integration on the UV-Vis spectral data corrected in Step S2, select univariate linear regression and XGBoost models for modeling, adjust hyperparameters on the training set, and test the model on the test set. Select the test set R. 2 The optimal parameter model with a value greater than 0.9; Step S4: Input the UV-Vis spectra from the test set in Step S3 into the trained optimal parameter model to obtain the organic matter concentration results of the wastewater.

[0008] Furthermore, the multivariate scattering correction MSC preprocessing described in step S2 includes the following steps: (1.1) Calculate the absorbance of the reference spectrum: , in, The absorbance is for reference spectrum. y ij (λ) represents the absorbance of the i-th sample at the j-th wavelength; n represents the number of wavelengths; λ represents wavelength, in nm; (1.2) Absorbance of the original spectrum y ij (λ) and absorbance of the reference spectrum Perform linear regression and solve for a using the least squares method. i and b i ; , , , Among them, a i b is a multiplicative factor i For additive shift, ε i (λ) represents the residual, a represents the intercept, and b represents the slope; (1.3) Using regression coefficients to analyze the original spectral absorbance y ij (λ) is corrected: , The absorbance of the i-th sample at the j-th wavelength after MSC correction is given.

[0009] Furthermore, in step S3, the univariate linear regression model is a linear regression model of the spectral integral value of a specific band of the ultraviolet-visible spectral data and the concentration of organic matter: , , Where S is the spectral integral value for a specific wavelength band, k is the slope, c is the intercept, C is the concentration of organic matter in wastewater (mg / L), λ0 is the incident wavelength (nm), and λ n The emission wavelength is in nm.

[0010] Furthermore, the XGBoost model modeling method described in step S3 includes the following steps: (1) Use the integral values ​​of different band intervals as input features; (2) Organic matter concentration prediction is achieved through model training, and the concentration value is output: , Where K is the number of trees, f k (x) is the predicted value of the k-th tree for sample x; (3) Optimize the model and identify the optimal band interval. Identify the optimal band interval by quantifying the contribution of each input feature in the model, and calculate the contribution of the input features according to the gain value: , Gain j N represents the gain value of feature j; splits The number of splits is represented by ΔS; the amount of loss reduction after splitting is represented by ΔS; and N represents the Nth split. The model is retrained using the band interval with the highest gain value to improve prediction efficiency and accuracy.

[0011] Furthermore, the multi-band segmented integration in step S3 is a band integration within any interval of 200-900nm, with a wavelength interval of 1 nm.

[0012] The beneficial effects of the technical solution provided by the embodiments of the present invention are as follows: (1) This invention addresses the technical bottlenecks of traditional organic matter concentration detection methods, such as time-consuming processes and the risk of secondary pollution. With the goal of achieving online real-time detection of organic matter concentration, reducing operating costs, and improving environmental benefits, it innovatively constructs an online organic matter concentration detection method based on "pretreatment-integral feature extraction-deep modeling". This solves the industry pain points of traditional methods, such as weak anti-interference ability, serious lag, and high cost. It provides a reliable technical guarantee for the refined management of wastewater treatment processes and the realization of energy conservation and emission reduction goals, and has significant economic benefits and social and ecological value.

[0013] (2) The MSC turbidity compensation method provided by the present invention can significantly improve the chemical information extraction capability of spectral data, is suitable for the analysis of complex samples, significantly improves the resistance to turbidity interference, and provides reliable input for subsequent modeling.

[0014] (3) By performing multi-band segmented integration on the corrected ultraviolet-visible spectrum, this invention can not only cover more organic absorption characteristics and provide feature contributions from different bands, but also reduce redundant information and avoid single wavelength (such as UV) 254 It significantly optimizes the utilization of spectral information and improves prediction accuracy by addressing the response blind zone of complex matrices. Attached Figure Description

[0015] Figure 1 This is a flowchart of a method for rapid online detection of organic matter concentration in wastewater based on deep learning, as described in an embodiment of the present invention.

[0016] Figure 2 This is the ultraviolet-visible spectrum of a wastewater sample from a wastewater treatment plant in an embodiment of the present invention.

[0017] Figure 3 This is a pre-processed ultraviolet-visible spectrum of a wastewater sample from a wastewater treatment plant in an embodiment of the present invention.

[0018] Figure 4 This is a simulation diagram of an exemplary univariate linear regression model in this embodiment. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] Example 1

[0021] like Figure 1 As shown, a rapid online detection method for organic matter concentration in wastewater based on deep learning includes the following steps: Step S1: Collect wastewater organic matter concentration data and UV-Vis spectra at different times to ensure correspondence between the organic matter concentration data and UV-Vis spectral data. Randomly divide the complete dataset into a training set (20%) and a test set (80%). Collect wastewater organic matter concentration data and UV-Vis spectra at different times to ensure correspondence between the organic matter concentration data and UV-Vis spectral data. The UV-Vis spectral data shown in this example is the training set spectrum, selecting data in the 200-900nm band with a sampling interval of 1 nm. Figure 2 As shown; Step S2: Preprocess the UV-Vis spectral data from Step S1. Since suspended particles in the water sample cause light scattering, raising the overall absorbance baseline of the UV-Vis spectral data and causing spurious absorption, compensating for turbidity in the water sample can significantly improve the data prediction results. Multivariate scattering correction (MSC) preprocessing is a very powerful tool for turbidity compensation, especially suitable for scenarios requiring non-destructive and rapid analysis of turbid liquids. Therefore, the UV-Vis spectral data (including the training and test sets) from Step S1 is preprocessed using MSC to obtain turbidity-compensated UV-Vis spectral data, such as... Figure 3 As shown, the specific steps include: (1.1) Calculate the absorbance of the reference spectrum: , in, The absorbance is for reference spectrum. y ij (λ) represents the absorbance of the i-th sample at the j-th wavelength; n represents the number of wavelengths; λ represents wavelength, in nm; (1.2) Absorbance of the original spectrum y ij (λ) and absorbance of the reference spectrum Perform linear regression and solve for a using the least squares method. i and b i ; , , , Among them, a i b is a multiplicative factor i For additive shift, ε i (λ) represents the residual, a represents the intercept, and b represents the slope; (1.3) Using regression coefficients to analyze the original spectral absorbance y ij (λ) is corrected: , The absorbance of the i-th sample at the j-th wavelength after MSC correction is given.

[0022] Current methods for online detection of organic matter concentration mostly rely on absorbance at a single wavelength or extract absorbance at only a few characteristic wavelengths, such as UV254 and UV280. Using only a few characteristic wavelength absorbance values ​​to predict organic matter concentration is difficult to fully cover all the information of organic matter, thus resulting in high prediction error of organic matter concentration. Therefore, the biggest improvement in this embodiment is to correlate the integral value of the continuous spectrum with the organic matter concentration.

[0023] Step S3: Integrate the UV-Vis spectral data in the training set after correction in step S2. In this embodiment, the integration is performed in the 200nm-900nm band. The integral equation is: , In this embodiment, S is the spectral integral value in the 200nm-900nm band, λ0 is the incident wavelength in nm; λ n The emission wavelength is in nm; Furthermore, a univariate linear regression model was used to linearly fit the organic matter concentration and spectral integral values ​​of the wastewater samples. The univariate linear regression model used was as follows: , Where k is the slope, c is the intercept, and C is the concentration of organic matter in the wastewater, in mg / L.

[0024] In this embodiment, the fitting result is as follows: Figure 4 As shown, the organic matter concentration exhibits a good linear relationship with the spectral integral value in the 200nm-900nm wavelength band. The linear regression model is: C = 0.875S - 14.95, where the R-squared value of the model fit is... 2 The value is 0.953. Therefore, the integral value of the continuous spectrum has a good correlation with the concentration of organic matter, and the concentration of organic matter can be detected using the spectral integral value. Moreover, this method is fast, environmentally friendly, and non-destructive, enabling online quantitative analysis of wastewater. At the same time, it can avoid the response blind zone of a single wavelength to complex matrices, significantly optimize the utilization rate of spectral information, and improve prediction accuracy.

[0025] Step S3 further models the UV-Vis spectral data after multivariate scattering correction MSC preprocessing using the XGBoost model. The specific method is as follows: (1) Adjust the hyperparameters of the training set, use the integral values ​​of different band intervals as input features, and achieve organic matter concentration prediction through model training, outputting the concentration value: , Where K is the number of trees, f k (x) is the predicted value of the k-th tree for sample x; (2) Model optimization: In this embodiment, the most important aspect is determining which band's integral value yields the highest accuracy in predicting organic matter concentration. Therefore, a gain function is introduced. Throughout the entire model (all trees), the gain is calculated each time the integral value corresponding to a different band is used. The higher the gain value for a particular band, the more likely the XGBoost model considers that band's integral value to be the most suitable for predicting organic matter concentration. Therefore, by quantifying the contribution of each feature (band integral value) in the model, key integration intervals are identified. The importance of the input features is calculated, and the integration intervals are sorted by gain. , Gain j N represents the gain value of feature j; splits The number of splits is represented by ΔS; the amount of loss reduction after splitting is represented by ΔS; and N represents the Nth split. The larger the gain, the more critical the integral interval is to improving the model's prediction accuracy. Eliminate the integral interval values ​​with lower ranking to simplify the model. The model is retrained using only the identified key bands to improve prediction efficiency and accuracy, and the model is tested on the test set.

[0026] Step S4: Select test set R 2 The optimal parameter model with a value greater than 0.9 is used to obtain the predicted value of organic matter concentration.

[0027] In this embodiment, the selected different wavelength bands are 200nm-400nm, 200nm-500nm, 200nm-600nm, 200nm-700nm, 200nm-800nm, 200nm-900nm, 220nm-400nm, 220nm-500nm, 220nm-600nm, 220nm-700nm, 220nm-800nm, and 220nm-900nm.

[0028] This invention eliminates suspended solids interference by establishing a turbidity compensation model, significantly improving resistance to turbidity interference and enhancing testing accuracy under complex water quality conditions. It extracts spectral integral features to construct a multi-band feature set and innovatively establishes a phased validation model from simple to complex. Univariate linear regression and XGBoost models are used to model the multi-band integral feature set and organic matter concentration, significantly optimizing spectral information utilization and avoiding reliance on single wavelengths (such as UV). 254 ) Response blind zone to complex matrices.

[0029] By constructing the XGBoost model, rapid and accurate prediction of organic matter concentration was achieved. Based on this, the selection of a single band integral value was broken, and the correlation between the segmented integral values ​​of different bands and organic matter concentration was established. This allows for more effective screening of the optimal band and improves the accuracy of prediction.

[0030] This invention addresses the industry pain points of traditional organic matter detection methods, such as weak anti-interference ability, serious lag, and high cost. It provides reliable technical support for the refined management of wastewater treatment processes and the achievement of energy conservation and emission reduction goals, and has significant economic benefits and social and ecological value.

[0031] This invention innovatively provides a phased verification model from simple to complex, and the specific embodiments described above are only preferred embodiments of this invention. It should be noted that for those skilled in the art, several improvements or substitutions can be made without departing from the principle of this invention, and these improvements or substitutions should be considered within the scope of protection of this invention.

Claims

1. A method for rapid online detection of organic matter concentration in wastewater based on deep learning, characterized in that, Includes the following steps: Step S1: Collect wastewater organic matter concentration data and ultraviolet-visible spectrum at different times to make the organic matter concentration data correspond to the ultraviolet-visible spectrum data, and divide the complete dataset into training set and test set; Step S2: Perform multivariate scattering correction (MSC) preprocessing on the UV-Vis spectral data described in step S1 to obtain turbidity-compensated UV-Vis spectral data. Step S3: Perform multi-band piecewise integration on the UV-Vis spectral data corrected in Step S2, select univariate linear regression and XGBoost models for modeling, adjust hyperparameters on the training set, and test the model on the test set. Select the test set R. 2 The optimal parameter model with a value greater than 0.9; Step S4: Input the UV-Vis spectra from the test set in Step S3 into the trained optimal parameter model to obtain the organic matter concentration results of the wastewater.

2. The method for rapid online detection of organic matter concentration in wastewater based on deep learning according to claim 1, characterized in that, The multivariate scattering correction MSC preprocessing described in step S2 includes the following steps: (1.1) Calculate the absorbance of the reference spectrum: ， in, The absorbance is for reference spectrum. y ij (λ) represents the absorbance of the i-th sample at the j-th wavelength; n represents the number of wavelengths; λ represents wavelength, in nm; (1.2) Absorbance of the original spectrum y ij (λ) and absorbance of the reference spectrum Perform linear regression and solve for a using the least squares method. i and b i ; ， ， ， Among them, a i b is a multiplicative factor i For additive shift, ε i (λ) represents the residual, a represents the intercept, and b represents the slope; (1.3) Using regression coefficients to analyze the original spectral absorbance y ij (λ) is corrected: ， The absorbance of the i-th sample at the j-th wavelength after MSC correction is given.

3. The method for rapid online detection of organic matter concentration in wastewater based on deep learning according to claim 1, characterized in that, In step S3, the univariate linear regression model is a linear regression model between the spectral integral value of a specific band of the ultraviolet-visible spectral data and the concentration of organic matter: ， ， Where S is the spectral integral value for a specific wavelength band, k is the slope, c is the intercept, C is the concentration of organic matter in wastewater (mg / L), λ0 is the incident wavelength (nm), and λ n The emission wavelength is in nm.

4. The method for rapid online detection of organic matter concentration in wastewater based on deep learning according to claim 1, characterized in that, The XGBoost model modeling method described in step S3 includes the following steps: (1) Use the integral values ​​of different band intervals as input features; (2) Organic matter concentration prediction is achieved through model training, and the concentration value is output: ， Where K is the number of trees, f k (x) is the predicted value of the k-th tree for sample x; (3) Optimize the model and identify the optimal band interval. Identify the optimal band interval by quantifying the contribution of each input feature in the model, and calculate the contribution of the input features according to the gain value: ， Gain j N represents the gain value of feature j; splits The number of splits is represented by ΔS; the amount of loss reduction after splitting is represented by ΔS; and N represents the Nth split. The model is retrained using the band interval with the highest gain value to improve prediction efficiency and accuracy.

5. The method for rapid online detection of organic matter concentration in wastewater based on deep learning according to claim 1, characterized in that, The multi-band segmented integration mentioned in step S3 is a band integration of any interval in the range of 200-900nm, with a wavelength interval of 1 nm.

Images