Online Monitoring Method for Water Quality Parameters Based on Deep Learning and Continuous Spectrum

Abstract

The present invention discloses an online monitoring method for water quality parameters based on deep learning and continuous spectra, which relates to the field of spectral analysis. The present invention includes: collecting spectral data using a spectrometer, and synchronously collecting data labels for supervised learning labels; constructing a deep learning model, selecting a model according to the spectral wavelength characteristics corresponding to the supervised learning labels, and optimizing the deep learning model by adding transfer learning; loading the optimized deep learning model into edge computing, and simultaneously performing real-time streaming processing on the spectral data of the next sampling period, and dynamically calibrating the model offset of the streaming processing according to the feedback data of the integrated electrochemical sensor; combining the isolation forest or the reconstruction error of the autoencoder to obtain an over-standard alarm for pollutants. The transfer learning of the present invention improves the generalization ability of the model, improves the data processing efficiency, and the shared layer enhances the suppression of common noise.

Description

Technical Field

[0001] The present invention relates to the field of spectral analysis, and specifically to an on-line monitoring method for water quality parameters based on deep learning and continuous spectra. Background Art

[0002] Spectral data may be affected by environmental factors, reducing the robustness of the deep learning model. At the same time, due to the problem of insufficient data, there is less data for certain rare pollution events, affecting the accuracy of real data.

[0003] In existing methods, the real-time requirements of on-line monitoring cannot be met. Due to the high complexity of the model structure, spectral noise interference, the water sample is not pre-treated, resulting in filtering particulate matter interfering with the spectral signal, and improper data feature selection, resulting in too high a dimension of important wavelength data and a fitting degree exceeding the preset expectation.

[0004] Moreover, to ensure on-line real-time data monitoring, there is a situation where there is less on-site data and more laboratory data, delaying the output of water quality parameter results in practical applications, with low efficiency and unable to provide guarantee for subsequent water quality safety. Summary of the Invention

[0005] The present invention provides an on-line monitoring method for water quality parameters based on deep learning and continuous spectra to solve the problems of the prior art.

[0006] In a first aspect, an on-line monitoring method for water quality parameters based on deep learning and continuous spectra includes:

[0007] Using a spectrometer to collect spectral data and synchronously collecting labels of the data for supervised learning labels;

[0008] Constructing a deep learning model, selecting a model according to the spectral wavelength characteristics corresponding to the supervised learning labels, and optimizing the deep learning model by adding transfer learning;

[0009] Loading the optimized deep learning model into edge computing, and at the same time, performing real-time streaming processing on the spectral data of the next sampling period, and dynamically calibrating the model offset of the streaming processing according to the feedback data of the integrated electrochemical sensor;

[0010] Combining the isolated forest or the reconstruction error of the autoencoder to obtain an over-standard alarm for pollutants.

[0011] Further, the using a spectrometer to collect spectral data and synchronously collecting labels of the data for supervised learning labels includes:

[0012] Adjusting the sampling frequency of the spectrometer in real time according to the change speed of water quality;

[0013] For each batch of spectral data, comparing with the sampling results of the laboratory to obtain the labels corresponding to the spectral data.

[0014] Furthermore, it also includes preprocessing the spectral data sampled by the spectrometer, including: smoothing high-frequency noise based on Savitzky-Golay filtering, selecting Daubechies wavelet basis, performing threshold denoising, and then selecting a baseline correction algorithm to separate the baseline and the effective spectrum by iteratively adjusting the penalty weight.

[0015] Furthermore, it also includes normalizing the preprocessed samples to eliminate the influence of optical path differences, and based on LASSO regression, loading L1 regularization to screen the wavelengths sensitive to the target parameters, where the target parameters are the water quality parameters corresponding to the feature selection.

[0016] Furthermore, when constructing the deep learning model, select the model according to the spectral wavelength characteristics corresponding to the supervised learning label, and optimize the deep learning model by adding transfer learning, including:

[0017] Select the corresponding deep learning model according to the water quality parameter characteristics, load the 1D-CNN model about the associated parameters of COD and BOD, regard the spectrum as a one-dimensional signal, extract local features through the convolutional layer, input the spectrum length, capture the wide absorption peak wavelengths according to the large kernel of the convolutional layer, extract local features according to the small kernel of the convolutional layer, and output the corresponding COD and BOD values.

[0018] Furthermore, after the CNN extracts features, connect it to the LSTM to model the time series, or after the CNN extracts features, combine it with the autoencoder for unsupervised pre-training to calculate the global correlation between wavelength points.

[0019] Furthermore, when constructing the deep learning model, select the model according to the spectral wavelength characteristics corresponding to the supervised learning label, and optimize the deep learning model by adding transfer learning, including:

[0020] Load adversarial training to align the spectral feature distributions of the target parameters in the laboratory and the field water area;

[0021] For the deep learning model regarding the COD water quality parameter, the shared underlying CNN extracts the common spectral features, performs sub-task prediction of different parameters on the top convolutional layer, the top convolutional layer is the TP prediction layer, and then performs fine-tuning of the prediction data on the fully connected layer. After fine-tuning, cycle through and input the preprocessed spectrometer sampling data of the next cycle until the data volume meets the preset medium data volume requirement, gradually unfreeze the intermediate convolutional layer. When the data volume meets the sufficient data volume requirement, adjust the weights of all convolutional layers, and synchronously increase the learning rate according to the time sequence of the data volume, from the pre-training layer to the subsequent training layer.

[0022] Furthermore, load edge computing into the optimized deep learning model, simultaneously perform real-time streaming processing on the spectral data of the next sampling cycle, and dynamically calibrate the model offset of the streaming processing according to the feedback data of the integrated electrochemical sensor. Specifically, it includes:

[0023] Among them, the electrochemical sensor outputs the pH meter measurement value. When the deviation between the predicted value of the deep learning model and the pH meter measurement value is > 10%, recalibration is triggered;

[0024] Embed a lightweight model in the spectrometer, process the spectral data stream in real time based on Apache Kafka or Flink, call the lightweight model for inference. When the sampling period of a specific water quality parameter is higher than that of other water quality parameters, set the deep learning model of the characteristic water quality parameter on the terminal device near the sensing element, and call the idle computing resources of the terminal to execute model inference.

[0025] Furthermore, for the construction of the deep learning model, select the model according to the spectral wavelength characteristics corresponding to the supervised learning labels, and optimize the deep learning model by adding transfer learning. It also includes: adding specific shallow convolutions corresponding to the water quality parameters after the shared bottom layer to capture the key features of the water quality parameters, and the key features of the water quality parameters are not included in the features of the shared layer;

[0026] Add a dynamic weight mechanism. By designing a loss function, assign weights to different water quality parameters to balance the dimensions. At the same time, automatically optimize the weights based on uncertainty weighting. An SE module is also set up to enhance the wavelength weights corresponding to the key features of the water quality parameters through the attention mechanism.

[0027] Furthermore, combine the Isolation Forest or the reconstruction error of the autoencoder to obtain the over-standard alarm of pollutants. Specifically, it includes: for the Isolation Forest, identify outliers based on the path length difference of spectral features; for the autoencoder, use it to train the AE to reconstruct the normal spectrum, calculate the reconstruction error as the anomaly score, and output the false alarm rate of the over-standard alarm of pollutants based on the anomaly score.

[0028] The online water quality parameter monitoring method based on deep learning and continuous spectrum provided by the present invention uses transfer learning to improve the generalization ability of the model and the data processing efficiency when there are few labeled sampling data. And due to the existence of the shared layer, the CNN model learns unified spectral preprocessing, enhancing the suppression of common noise.

[0029] The present invention performs transfer learning to optimize the deep learning model, avoiding repeated extraction of underlying features for each task, reducing the computational amount, and enhancing the number of parameters predicted simultaneously during a single spectral scan in real-time monitoring. Description of the Drawings

[0030] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, form a part of the present invention, and do not limit the embodiments of the present invention. In the drawings:

[0031] Figure 1Flowchart of an online water quality parameter monitoring method based on deep learning and continuous spectrum provided by an exemplary embodiment of the present invention. Detailed implementation manners

[0032] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present invention. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0033] First, the nouns involved in the present invention are explained:

[0034] Continuous spectrum measurement data, by continuously collecting spectral data of water samples through, for example, ultraviolet-visible spectroscopy and near-infrared spectroscopy, including absorbance or reflectance at different wavelengths, can be used to analyze various components in water.

[0035] Water quality parameters, including COD, BOD, ammonia nitrogen, total phosphorus, heavy metals, etc.

[0036] To perform deep learning, the required process steps are: data acquisition, preprocessing, feature extraction, model training, online monitoring system design, and verification and optimization.

[0037] Specifically, due to the common characteristics of spectral data, that is, in the spectral curves of different water quality parameters (such as COD, BOD, NH3-N), some basic characteristics may be shared, for example:

[0038] Absorption peak position (for example, the nitrate absorption peak at 220 nm may affect the correlation of multiple parameters);

[0039] Baseline shape (the scattering effect caused by turbidity may simultaneously interfere with the measurement of multiple parameters);

[0040] Noise pattern (environmental interference has similarities in the spectra of different parameters)

[0041] And the role of the underlying CNN: automatically extract these cross-parameter common characteristics through the convolutional layer (Conv1D), such as edge detection (identifying absorption peaks), local pattern matching (finding combinations of characteristic peaks), etc.

[0042] Therefore, the inventive concept solves the problem of poor real-time performance caused by less data through the top-level CNN sub-task head, combines features with parameter specificity, COD: depends on the absorbance combination of multiple wavelengths, such as 254nm, 350nm, turbidity: related to the scattered light intensity of the full wavelength band. Based on the shared features, through independent fully connected layers (Dense) or lightweight sub-networks, learn the parameter-specific non-linear mapping relationship. By setting the shared layer, force all tasks to share the underlying convolutional layer, and force the model to learn general spectral features, and then set independent heads, for example, set the COD prediction head, set the ammonia nitrogen prediction head, that is:

[0043] cod_output = Dense(32, activation='relu')(shared_features)

[0044] cod_output = Dense(1, name='cod')(cod_output)

[0045] nh3_output = Dense(64, activation='relu')(shared_features)

[0046] nh3_output = Dense(1, name='nh3')(nh3_output)

[0047] Let each task have an independent dense layer and allow parameter-specific non-linear transformation.

[0048] The specific application scenario of the present invention is real-time monitoring of water quality parameters.

[0049] The on-line water quality parameter monitoring method based on deep learning and continuous spectrum provided by the present invention aims to solve the above technical problems of the prior art.

[0050] The technical solution of the present invention and how the technical solution of the present invention solves the above technical problems will be described in detail below with specific embodiments. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below with reference to the accompanying drawings.

[0051] Embodiment 1: This embodiment provides an on-line water quality parameter monitoring method based on deep learning and continuous spectrum, including, as Figure 1 shown:

[0052] S1. Use a spectrometer to collect spectral data and synchronously collect the labels of the data for supervised learning of the labels;

[0053] Including: adjusting the sampling frequency of the spectrometer in real-time according to the water quality change rate; setting according to the water quality change rate (such as once per minute for river monitoring, once every 10 seconds for wastewater treatment plant process control).

[0054] For each batch of spectral data, compare with the sampling results of the laboratory to obtain the labels corresponding to the spectral data.

[0055] It also includes preprocessing the spectral data sampled by the spectrometer, including: smoothing high-frequency noise based on Savitzky-Golay filtering, selecting a window size of 7-15 points, a polynomial order of 2-3, selecting the Daubechies wavelet basis, decomposing to 3-5 layers, performing threshold denoising using the Birgé-Massart strategy, and then selecting the baseline correction algorithm, airPLS, and separating the baseline (i.e., low-frequency signal) and the effective spectrum (i.e., high-frequency signal) by iteratively adjusting the penalty weight.

[0056] It also includes normalizing the preprocessed samples to eliminate the influence of optical path differences, and based on LASSO regression, loading L1 regularization to screen the wavelengths sensitive to the target parameters, and the hyperparameter λ is determined by cross-validation. The target parameters are the water quality parameters corresponding to the feature selection.

[0057] Regarding the selection of spectrometer types, UV-Vis spectrometers (wavelength range 200-800 nm): suitable for detecting organic pollutants (such as COD, BOD), ammonia nitrogen (NH3-N), etc., based on specific absorption peaks (such as the absorption of nitrate at 220 nm). NIR spectrometers (700-2500 nm): suitable for detecting oils, sugars, suspended solids (turbidity) in water, using the vibration harmonics of O-H and C-H bonds. Raman spectrometers: used to detect the fingerprint peaks of heavy metal ions (such as Cr 6 ⁺, Pb²⁺), but fluorescence interference needs to be suppressed (such as using surface-enhanced Raman technology SERS).

[0058] S2. Build a deep learning model, select the model according to the spectral wavelength characteristics corresponding to the supervised learning labels, and optimize the deep learning model by adding transfer learning.

[0059] Including: selecting the corresponding deep learning model according to the water quality parameter characteristics, loading the 1D-CNN model regarding the associated parameters of COD and BOD, regarding the spectrum as a one-dimensional signal, extracting local features through the convolutional layer, inputting the spectrum length, capturing the wide absorption peak wavelengths according to the large kernel of the convolutional layer, extracting local features according to the small kernel of the convolutional layer, and outputting the corresponding COD and BOD values. After the CNN extracts features, connect to the LSTM to model the time series for predicting the change trend of water quality parameters over time, or after the CNN extracts features, combine with the autoencoder for unsupervised pre-training to calculate the global correlation between wavelength points. Load adversarial training to align the spectral feature distributions of the target parameters in the laboratory and the on-site water area.

[0060] The following is an example of a 1D-CNN structure:

[0061] model = Sequential(

[0062] Input(shape=(n_wavelengths, 1)), # Input spectral length (e.g., 500 wavelength points)

[0063] Conv1D(filters=64, kernel_size=15,activation='relu'), # Large kernel captures wide absorption peaks

[0064] MaxPooling1D(pool_size=2),

[0065] Conv1D(filters=128, kernel_size=5,activation='relu'), # Small kernel extracts local features

[0066] GlobalAveragePooling1D(),

[0067] Dense(128, activation='relu'),

[0068] Dense(1) # Output COD value )

[0070] For the deep learning model regarding COD water quality parameters, the underlying CNN will share and extract common spectral features. The top convolutional layer will perform sub-task predictions for different parameters. The top convolutional layer is the TP prediction layer. Then, the fully connected layer will perform fine-tuning on the predicted data. After fine-tuning, the pre-processed spectrometer sampling data of the next cycle will be cyclically input until the data volume meets the preset medium data volume requirement. Then, the intermediate convolutional layers will be gradually unfrozen. When the data volume meets the sufficient data volume requirement, the weights of all convolutional layers will be adjusted. Along with the time sequence of the data volume, from the pre-trained layer to the subsequent training layers, the learning rate will be synchronously increased. It also includes: adding specific shallow convolutions corresponding to the water quality parameters after the shared underlying layer to capture the key features of the water quality parameters, and the key features of the water quality parameters are not included in the features of the shared layer; since the laboratory pure water sample dataset has high-precision labels and is used to train the shared layer, the shared layer has general spectral features, enhancing the suppression of common noise.

[0071] A dynamic weight mechanism is added. By designing the loss function, weight balance dimensions are assigned to different water quality parameters. At the same time, based on uncertainty weighting, the weights are automatically optimized. An SE module is also set up to enhance the wavelength weights corresponding to the key features of the water quality parameters through the attention mechanism.

[0072] S3. Load the optimized deep learning model into edge computing, and simultaneously perform real-time streaming processing on the spectral data of the next sampling period, and dynamically calibrate the model offset of the streaming processing according to the feedback data of the integrated electrochemical sensor;

[0073] Among them, the electrochemical sensor outputs the measured value of the pH meter. When the deviation between the predicted value of the deep learning model and the measured value of the pH meter > 10%, recalibration is triggered;

[0074] Embed a lightweight model in the spectrometer, process the spectral data stream in real time based on Apache Kafka or Flink, call the lightweight model for inference. When the sampling period of specific water quality parameters is higher than that of other water quality parameters, set the deep learning model of the characteristic water quality parameters on the terminal device near the sensing element, and call the idle computing resources of the terminal to perform model inference.

[0075] S4. Combine the Isolation Forest or the reconstruction error of the autoencoder to obtain the over-standard alarm of pollutants.

[0076] In several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0077] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0078] In addition, in each embodiment of the present invention, the functional modules can be integrated in a processing module, or each module can exist physically alone, or two or more modules can be integrated in one module. The above integrated modules can be implemented in the form of hardware, or in the form of a combination of hardware and software function modules.

[0079] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method or a system. Therefore, the present invention can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects.

[0080] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity or device comprising a series of elements not only includes those elements but also other elements not expressly listed, or elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising the element.

[0081] The above are only embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, various changes and modifications can be made to the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the scope of the claims of the present invention.

[0082] After considering the specification and practicing the invention disclosed herein, those skilled in the art will readily conceive of other embodiments of the present invention. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common general knowledge or conventional technical means in the technical field not disclosed by the present invention. The specification and examples are only regarded as exemplary, and the true scope and spirit of the present invention are pointed out by the claims above.

[0083] It should be understood that the present invention is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. An online monitoring method for water quality parameters based on deep learning and continuous spectrum, characterized in that: include: A spectrometer is used to collect spectral data, and labels of the collected data are used for supervised learning labels; Construct a deep learning model, select a model based on the spectral wavelength features corresponding to the supervised learning label, and optimize the deep learning model by adding transfer learning, including: select the corresponding deep learning model based on the water quality parameter features, load the 1D-CNN model of COD and BOD correlation parameters, regard the spectrum as a one-dimensional signal, extract local features through the convolution layer, input the spectrum length, capture the wide absorption peak wavelength based on the large kernel of the convolution layer, extract local features based on the small kernel of the convolution layer, and output the corresponding COD and BOD values; connect the LSTM modeling sequence after extracting features from CNN, or perform unsupervised pre-training with the autoencoder after extracting features from CNN to calculate the wavelength points. Global correlation; load adversarial training, align the spectral feature distribution of the target parameters of the laboratory and field waters; for the deep learning model of COD water quality parameters, share the underlying CNN to extract the common spectral features, perform sub-task prediction of different parameters on the top convolution layer, the top convolution layer is the TP prediction layer, and then perform prediction data fine-tuning on the fully connected layer. After fine-tuning, the pre-processed spectrometer sampling data of the next cycle is cyclically entered until the data volume meets the preset medium data volume requirement, and the intermediate convolution layers are gradually unfrozen. When the data volume meets the sufficient data volume requirement, adjust the weights of all convolution layers, and increase the learning rate synchronously from the pre-training layer to the subsequent training layer in the order of data volume; The optimized deep learning model is loaded into edge computing, and the spectral data of the next sampling period is streamed in real time, and the model offset of the streaming processing is dynamically calibrated according to the feedback data of the integrated electrochemical sensor; Combined with isolation forest or autoencoder reconstruction error, an alarm for exceeding the pollutant standard is obtained.

2. The method for online monitoring of water quality parameters based on deep learning and continuous spectrum according to claim 1, characterized in that: The method of using a spectrometer to collect spectral data and synchronously collecting labels of the data for supervised learning labels includes: Adjust the sampling frequency of the spectrometer in real time according to the speed of water quality change; For each batch of spectral data, the label of the corresponding spectral data is obtained by comparing it with the sampling results of the laboratory.

3. The method for online monitoring of water quality parameters based on deep learning and continuous spectrum according to claim 2 is characterized in that: It also includes preprocessing of the spectral data sampled by the spectrometer, including: smoothing high-frequency noise based on Savitzky-Golay filtering, selecting Daubechies wavelet basis, performing threshold denoising, and then selecting a baseline correction algorithm to separate the baseline from the effective spectrum by iteratively adjusting the penalty weights.

4. The method for online monitoring of water quality parameters based on deep learning and continuous spectrum according to claim 3 is characterized in that: It also includes normalizing the pre-processed samples to eliminate the influence of optical path differences, and based on LASSO regression, loading L1 regularization to screen wavelengths sensitive to target parameters, where the target parameters are water quality parameters selected according to the corresponding features.

5. The method for online monitoring of water quality parameters based on deep learning and continuous spectrum according to claim 4 is characterized in that: The optimized deep learning model is loaded into edge computing, and the spectral data of the next sampling period is streamed in real time, and the model offset of the streaming processing is dynamically calibrated according to the feedback data of the integrated electrochemical sensor. include: Among them, the electrochemical sensor outputs the pH meter measurement value. When the deviation between the deep learning model prediction value and the pH meter measurement value is greater than 10%, recalibration is triggered; Embed a lightweight model in the spectrometer, process the spectral data stream in real time based on Apache Kafka or Flink, and call the lightweight model for reasoning. When the sampling period of a specific water quality parameter is higher than that of other water quality parameters, set the deep learning model of the characteristic water quality parameter on the terminal device near the sensor, and call the idle computing resources of the terminal to perform model reasoning.

6. The method for online monitoring of water quality parameters based on deep learning and continuous spectrum according to claim 5 is characterized in that: The deep learning model is constructed, a model is selected according to the spectral wavelength characteristics corresponding to the supervised learning label, and the deep learning model is optimized by adding transfer learning, and further includes: adding a shallow convolution corresponding to the water quality parameter specific to the shared bottom layer to capture the key characteristics of the water quality parameter, and the key characteristics of the water quality parameter are not included in the characteristics of the shared layer; A dynamic weight mechanism is added to assign weight balance dimensions to different water quality parameters by designing a loss function. At the same time, the weights are automatically optimized based on uncertainty weighting. A SE module is also provided to enhance the wavelength weights corresponding to the key features of the water quality parameters through an attention mechanism.

7. The method for online monitoring of water quality parameters based on deep learning and continuous spectrum according to claim 6 is characterized in that: The method of combining the isolation forest or autoencoder reconstruction error to obtain the pollutant exceeding the standard alarm specifically includes: the isolation forest identifies outliers based on the path length difference of the spectral characteristics; for the autoencoder, it is used to train AE to reconstruct the normal spectrum, calculate the reconstruction error as the anomaly score, and output the false alarm rate of the pollutant exceeding the standard alarm based on the anomaly score.

Images