Water quality parameter online monitoring method based on deep learning and continuous spectrum

Abstract

The invention discloses a water quality parameter on-line monitoring method based on deep learning and continuous spectrum, and relates to the field of spectrum analys.The method comprises the steps that a spectrograph is adopted to collect spectrum data, and labels of the data are synchronously collected to be used for supervising and learning the labels; constructing a deep learning model, selecting a model according to the spectral wavelength characteristics corresponding to the supervised learning label, and optimizing the deep learning model by adding transfer learning; loading edge calculation into the optimized deep learning model, performing real-time streaming processing on the spectral data of the next sampling period, and dynamically calibrating the model offset of the streaming processing according to the feedback data of the integrated electrochemical sensor; according to the method, by means of transfer learning, the generalization ability of the model is improved, the data processing efficiency is improved, and the suppression of common noise is enhanced by the sharing layer.

Description

Technical Field

[0001] The present invention relates to the field of spectral analysis, and in particular to an online monitoring method for water quality parameters based on deep learning and continuous spectrum. Background Art

[0002] Spectral data may be affected by environmental factors, which reduces the robustness of deep learning models. At the same time, due to insufficient data, there is less data on some rare pollution events, which affects the accuracy of real data. The existing methods cannot meet the real-time requirements of online monitoring due to the high complexity of the model structure, spectral noise interference, untreated water samples resulting in filtered particles interfering with spectral signals, improper selection of data features resulting in excessively high dimensions of important wavelength data, and a degree of fit exceeding preset expectations.

[0003] Moreover, in order to ensure online real-time data monitoring, there is a situation where there is less field data and more laboratory data. In actual applications, the output of water quality parameter results is delayed, which is inefficient and cannot provide guarantees for subsequent water quality safety. Summary of the invention

[0004] The present invention provides an online monitoring method for water quality parameters based on deep learning and continuous spectrum, which solves the problems of the prior art.

[0005] First, the online monitoring method of water quality parameters based on deep learning and continuous spectrum includes: A spectrometer is used to collect spectral data, and labels of the collected data are used for supervised learning labels; Build a deep learning model, select the model based on the spectral wavelength features corresponding to the supervised learning labels, and optimize the deep learning model by adding transfer learning; The optimized deep learning model is loaded into edge computing, and the spectral data of the next sampling period is streamed in real time, and the model offset of the streaming processing is dynamically calibrated according to the feedback data of the integrated electrochemical sensor; Combined with isolation forest or autoencoder reconstruction error, an alarm for excessive pollutants can be obtained.

[0006] Furthermore, the method of using a spectrometer to collect spectral data and synchronously collecting labels of the data for supervised learning labels includes: Adjust the sampling frequency of the spectrometer in real time according to the speed of water quality change; For each batch of spectral data, the label of the corresponding spectral data is obtained by comparing it with the sampling results of the laboratory.

[0007] Furthermore, the method also includes preprocessing the spectral data sampled by the spectrometer, including: smoothing high-frequency noise based on Savitzky-Golay filtering, selecting Daubechies wavelet basis, performing threshold denoising, and then selecting a baseline correction algorithm to separate the baseline and the effective spectrum by iteratively adjusting the penalty weight.

[0008] Furthermore, it also includes normalizing the pre-processed samples to eliminate the influence of optical path differences, and based on LASSO regression, loading L1 regularization to screen wavelengths sensitive to target parameters, where the target parameters are water quality parameters selected by the corresponding features.

[0009] Furthermore, the deep learning model is constructed, a model is selected according to the spectral wavelength feature corresponding to the supervised learning label, and the deep learning model is optimized by adding transfer learning, including: Select the corresponding deep learning model according to the characteristics of water quality parameters, load the 1D-CNN model of COD and BOD correlation parameters, regard the spectrum as a one-dimensional signal, extract local features through the convolution layer, input the spectrum length, capture the wide absorption peak wavelength based on the large core of the convolution layer, extract local features based on the small core of the convolution layer, and output the corresponding COD and BOD values.

[0010] Furthermore, after extracting features from CNN, LSTM is connected to model the time series, or after extracting features from CNN, unsupervised pre-training is performed with an autoencoder to calculate the global correlation between wavelength points.

[0011] Furthermore, the deep learning model is constructed, a model is selected according to the spectral wavelength feature corresponding to the supervised learning label, and the deep learning model is optimized by adding transfer learning, including: Load adversarial training to align the spectral feature distributions of target parameters in the laboratory and field waters; For the deep learning model of COD water quality parameters, the shared underlying CNN is used to extract common spectral features, and the top convolutional layer is used to perform sub-task prediction of different parameters. The top convolutional layer is the TP prediction layer, and then the fully connected layer is used to perform prediction data fine-tuning. After fine-tuning, the pre-processed spectrometer sampling data of the next cycle is cyclically entered until the data volume meets the preset medium data volume requirement. The intermediate convolutional layers are gradually unfrozen. When the data volume meets the sufficient data volume requirement, the weights of all convolutional layers are adjusted, and the learning rate is synchronously increased from the pre-training layer to the subsequent training layer in chronological order of the data volume.

[0012] Furthermore, the optimized deep learning model is loaded into edge computing, and the spectral data of the next sampling period is streamed in real time, and the model offset of the streaming processing is dynamically calibrated according to the feedback data of the integrated electrochemical sensor, specifically including: Among them, the electrochemical sensor outputs the pH meter measurement value. When the deviation between the deep learning model prediction value and the pH meter measurement value is greater than 10%, recalibration is triggered; Embed a lightweight model in the spectrometer, process the spectral data stream in real time based on Apache Kafka or Flink, and call the lightweight model for reasoning. When the sampling period of a specific water quality parameter is higher than that of other water quality parameters, set the deep learning model of the characteristic water quality parameter on the terminal device near the sensor, and call the idle computing resources of the terminal to perform model reasoning.

[0013] Furthermore, the construction of the deep learning model, selecting the model according to the spectral wavelength features corresponding to the supervised learning labels, and optimizing the deep learning model by adding transfer learning, further includes: adding a shallow convolution corresponding to the water quality parameter specific to the shared bottom layer to capture the key features of the water quality parameters, the key features of the water quality parameters not included in the features of the shared layer; A dynamic weight mechanism is added to assign weight balance dimensions to different water quality parameters by designing a loss function. At the same time, the weights are automatically optimized based on uncertainty weighting. A SE module is also provided to enhance the wavelength weights corresponding to the key features of the water quality parameters through an attention mechanism.

[0014] Furthermore, the described method combines the isolation forest or autoencoder reconstruction error to obtain the pollutant exceeding standard alarm, specifically including: the isolation forest identifies outliers based on the path length difference of the spectral characteristics; for the autoencoder, it is used to train AE to reconstruct the normal spectrum, calculate the reconstruction error as the anomaly score, and output the false alarm rate of the pollutant exceeding standard alarm based on the anomaly score.

[0015] The online monitoring method for water quality parameters based on deep learning and continuous spectrum provided by the present invention uses transfer learning to enhance the generalization ability of the model and the data processing efficiency when the sampling data is less labeled. Moreover, due to the existence of the shared layer, the CNN model learns unified spectral preprocessing to enhance the suppression of common noise.

[0016] The present invention performs transfer learning to optimize the deep learning model, avoids repeated extraction of underlying features for each task, reduces the amount of calculation, and enhances the number of parameters predicted simultaneously in a single spectral scan in real-time monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of the present invention, and do not constitute a limitation of the embodiments of the present invention. In the drawings: Figure 1 A flow chart of a method for online monitoring of water quality parameters based on deep learning and continuous spectrum provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0018] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0019] First, the terms involved in the present invention are explained: Continuous spectral measurement data, such as ultraviolet-visible spectroscopy and near-infrared spectroscopy, continuously collects spectral data of water samples, including absorbance or reflectance at different wavelengths, which can be used to analyze various components in water.

[0020] Water quality parameters, including COD, BOD, ammonia nitrogen, total phosphorus, heavy metals, etc.

[0021] To perform deep learning, the process steps required are: data collection, preprocessing, feature extraction, model training, online monitoring system design, and verification and optimization.

[0022] Specifically, due to the commonality of spectral data, the spectral curves of different water quality parameters (such as COD, BOD, NH3-N) may share certain basic features, such as: Absorption peak position (e.g., the nitrate absorption peak at 220 nm may affect the correlation of multiple parameters); Baseline shape (scattering effects caused by turbidity may interfere with the measurement of multiple parameters simultaneously); Noise pattern (environmental disturbances have similarities in the spectra of different parameters) The role of the underlying CNN is to automatically extract these common features across parameters through the convolutional layer (Conv1D), such as edge detection (identifying absorption peaks), local pattern matching (discovering characteristic peak combinations), etc.

[0023] Therefore, the present invention is conceived to solve the problem of poor real-time performance caused by less data through the top-level CNN sub-task head, and to combine the parameter specificity into features. COD: depends on the absorbance combination of multiple wavelengths, such as 254nm, 350nm, turbidity: related to the scattered light intensity of the whole band. On the basis of shared features, through independent fully connected layers (Dense) or lightweight sub-networks, learn parameter-specific nonlinear mapping relationships. By setting a shared layer, all tasks are forced to share the underlying convolutional layer, and the model is forced to learn common spectral features, and then independent heads are set, for example, a COD prediction head is set, and an ammonia nitrogen prediction head is set, that is: cod_output = Dense(32, activation='relu')(shared_features) cod_output = Dense(1, name='cod')(cod_output) nh3_output = Dense(64, activation='relu')(shared_features) nh3_output = Dense(1, name='nh3')(nh3_output) Let each task have independent dense layers and allow parameter-specific nonlinear transformations.

[0024] The specific application scenario of the present invention is real-time monitoring of water quality parameters.

[0025] The online monitoring method of water quality parameters based on deep learning and continuous spectrum provided by the present invention is intended to solve the above technical problems of the prior art.

[0026] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0027] Embodiment 1: This embodiment provides a method for online monitoring of water quality parameters based on deep learning and continuous spectrum, including: Figure 1 As shown: S1, using a spectrometer to collect spectral data, and synchronously collect data labels for supervised learning labels; Including: adjusting the sampling frequency of the spectrometer in real time according to the speed of water quality change; setting according to the speed of water quality change (such as river monitoring 1 time / minute, sewage plant process control 1 time / 10 seconds) For each batch of spectral data, the label of the corresponding spectral data is obtained by comparing it with the sampling results of the laboratory.

[0028] It also includes preprocessing of the spectral data sampled by the spectrometer, including: smoothing high-frequency noise based on Savitzky-Golay filtering, selecting a window size of 7-15 points, a polynomial order of 2-3, selecting a Daubechies wavelet basis, decomposing it into 3-5 layers, and performing threshold denoising using the Birgé-Massart strategy. Then, the baseline correction algorithm, airPLS, is selected to iteratively adjust the penalty weights to separate the baseline, i.e., the low-frequency signal, and the effective spectrum, i.e., the high-frequency signal.

[0029] It also includes normalizing the pre-processed samples to eliminate the influence of optical path differences, and based on LASSO regression, loading L1 regularization to screen wavelengths sensitive to target parameters, and the hyperparameter λ is determined by cross-validation. The target parameter is the water quality parameter selected by the corresponding feature; Regarding the type of spectrometer, UV-Vis spectrometer (wavelength range 200-800nm): suitable for detecting organic pollutants (such as COD, BOD), ammonia nitrogen (NH3-N), etc., based on specific absorption peaks (such as nitrate absorption at 220nm). NIR spectrometer (700-2500nm): suitable for detecting oils, sugars, suspended matter (turbidity) in water, using the vibration harmonics of OH and CH bonds. Raman spectrometer: used to detect heavy metal ions (such as Cr 6 ⁺, Pb²⁺), but the fluorescence interference needs to be suppressed (such as using surface enhanced Raman technology SERS).

[0030] S2. Build a deep learning model, select the model based on the spectral wavelength features corresponding to the supervised learning labels, and optimize the deep learning model by adding transfer learning; Including: selecting the corresponding deep learning model according to the characteristics of water quality parameters, loading the 1D-CNN model of COD and BOD correlation parameters, treating the spectrum as a one-dimensional signal, extracting local features through the convolution layer, inputting the spectrum length, capturing the wide absorption peak wavelength based on the large core of the convolution layer, extracting local features based on the small core of the convolution layer, and outputting the corresponding COD and BOD values. After extracting features from CNN, connect the LSTM modeling sequence to predict the trend of water quality parameters over time, or extract features from CNN and combine it with the autoencoder for unsupervised pre-training to calculate the global correlation between wavelength points. Load adversarial training to align the spectral feature distribution of the target parameters in the laboratory and on-site waters; The following is an example of a 1D-CNN structure: model = Sequential([ Input(shape=(n_wavelengths, 1)),# Input spectrum length (such as 500 wavelength points) Conv1D(filters=64, kernel_size=15,activation='relu'),# Large kernel captures wide absorption peak MaxPooling1D(pool_size=2), Conv1D(filters=128, kernel_size=5,activation='relu'),# Small kernel extracts local features GlobalAveragePooling1D(), Dense(128, activation='relu'), Dense(1)# Output COD value ]) For the deep learning model of COD water quality parameters, the shared bottom layer CNN is used to extract the common features of the spectrum, and the top convolution layer is used to perform task prediction of different parameters. The top convolution layer is the TP prediction layer, and then the prediction data is fine-tuned for the fully connected layer. After fine-tuning, the pre-processed spectrometer sampling data of the next cycle is recorded in a loop until the data volume meets the preset medium data volume requirement, and the intermediate convolution layers are gradually unfrozen. When the data volume meets the sufficient data volume requirement, the weights of all convolution layers are adjusted, and the learning rate is synchronously increased from the pre-training layer to the subsequent training layer in the order of the data volume. It also includes: adding a shallow convolution corresponding to the water quality parameter specific to the shared bottom layer to capture the key features of the water quality parameter, which are not included in the features of the shared layer; since the laboratory pure water sample data set has high-precision labels, it is used to train the shared layer, so that the shared layer has universal spectral features to enhance the suppression of common noise.

[0031] A dynamic weight mechanism is added to assign weight balance dimensions to different water quality parameters by designing a loss function. At the same time, the weights are automatically optimized based on uncertainty weighting. A SE module is also provided to enhance the wavelength weights corresponding to the key features of the water quality parameters through an attention mechanism.

[0032] S3. Load the optimized deep learning model into edge computing, and process the spectral data of the next sampling period in real time, and dynamically calibrate the model offset of the streaming processing according to the feedback data of the integrated electrochemical sensor; Among them, the electrochemical sensor outputs the pH meter measurement value. When the deviation between the deep learning model prediction value and the pH meter measurement value is greater than 10%, recalibration is triggered; Embed a lightweight model in the spectrometer, process the spectral data stream in real time based on Apache Kafka or Flink, and call the lightweight model for reasoning. When the sampling period of a specific water quality parameter is higher than that of other water quality parameters, set the deep learning model of the characteristic water quality parameter on the terminal device near the sensor, and call the idle computing resources of the terminal to perform model reasoning.

[0033] S4. Combine the isolation forest or autoencoder reconstruction error to obtain the excessive pollutant alarm.

[0034] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0035] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0036] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0037] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods or systems. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects.

[0038] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0039] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

[0040] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention is indicated by the claims above.

[0041] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An online monitoring method for water quality parameters based on deep learning and continuous spectrum, characterized in that: include: A spectrometer is used to collect spectral data, and labels of the collected data are used for supervised learning labels; Build a deep learning model, select the model based on the spectral wavelength features corresponding to the supervised learning labels, and optimize the deep learning model by adding transfer learning; The optimized deep learning model is loaded into edge computing, and the spectral data of the next sampling period is streamed in real time, and the model offset of the streaming processing is dynamically calibrated according to the feedback data of the integrated electrochemical sensor; Combined with isolation forest or autoencoder reconstruction error, an alarm for exceeding the pollutant standard is obtained.

2. The method for online monitoring of water quality parameters based on deep learning and continuous spectrum according to claim 1, characterized in that: The method of using a spectrometer to collect spectral data and synchronously collecting labels of the data for supervised learning labels includes: Adjust the sampling frequency of the spectrometer in real time according to the speed of water quality change; For each batch of spectral data, the label of the corresponding spectral data is obtained by comparing it with the sampling results of the laboratory.

3. The method for online monitoring of water quality parameters based on deep learning and continuous spectrum according to claim 2 is characterized in that: It also includes preprocessing of the spectral data sampled by the spectrometer, including: smoothing high-frequency noise based on Savitzky-Golay filtering, selecting Daubechies wavelet basis, performing threshold denoising, and then selecting a baseline correction algorithm to separate the baseline from the effective spectrum by iteratively adjusting the penalty weights.

4. The method for online monitoring of water quality parameters based on deep learning and continuous spectrum according to claim 3 is characterized in that: It also includes normalizing the pre-processed samples to eliminate the influence of optical path differences, and based on LASSO regression, loading L1 regularization to screen wavelengths sensitive to target parameters, where the target parameters are water quality parameters selected according to the corresponding features.

5. The method for online monitoring of water quality parameters based on deep learning and continuous spectrum according to claim 4 is characterized in that: The deep learning model is constructed, a model is selected according to the spectral wavelength feature corresponding to the supervised learning label, and the deep learning model is optimized by adding transfer learning, including: Select the corresponding deep learning model according to the characteristics of water quality parameters, load the 1D-CNN model of COD and BOD correlation parameters, regard the spectrum as a one-dimensional signal, extract local features through the convolution layer, input the spectrum length, capture the wide absorption peak wavelength based on the large core of the convolution layer, extract local features based on the small core of the convolution layer, and output the corresponding COD and BOD values.

6. The method for online monitoring of water quality parameters based on deep learning and continuous spectrum according to claim 5 is characterized in that: After extracting features from CNN, connect it to LSTM to model the time series, or after extracting features from CNN, combine it with the autoencoder for unsupervised pre-training to calculate the global correlation between wavelength points.

7. The method for online monitoring of water quality parameters based on deep learning and continuous spectrum according to claim 6 is characterized in that: The deep learning model is constructed, a model is selected according to the spectral wavelength feature corresponding to the supervised learning label, and the deep learning model is optimized by adding transfer learning, including: Load adversarial training to align the spectral feature distributions of target parameters in the laboratory and field waters; For the deep learning model of COD water quality parameters, the shared underlying CNN is used to extract common spectral features, and the top convolutional layer is used to perform sub-task prediction of different parameters. The top convolutional layer is the TP prediction layer, and then the fully connected layer is used to perform prediction data fine-tuning. After fine-tuning, the pre-processed spectrometer sampling data of the next cycle is cyclically entered until the data volume meets the preset medium data volume requirement. The intermediate convolutional layers are gradually unfrozen. When the data volume meets the sufficient data volume requirement, the weights of all convolutional layers are adjusted, and the learning rate is synchronously increased from the pre-training layer to the subsequent training layer in chronological order of the data volume.

8. The method for online monitoring of water quality parameters based on deep learning and continuous spectrum according to claim 7 is characterized in that: The optimized deep learning model is loaded into edge computing, and the spectral data of the next sampling period is streamed in real time, and the model offset of the streaming processing is dynamically calibrated according to the feedback data of the integrated electrochemical sensor. include: Among them, the electrochemical sensor outputs the pH meter measurement value. When the deviation between the deep learning model prediction value and the pH meter measurement value is greater than 10%, recalibration is triggered; Embed a lightweight model in the spectrometer, process the spectral data stream in real time based on Apache Kafka or Flink, and call the lightweight model for reasoning. When the sampling period of a specific water quality parameter is higher than that of other water quality parameters, set the deep learning model of the characteristic water quality parameter on the terminal device near the sensor, and call the idle computing resources of the terminal to perform model reasoning.

9. The method for online monitoring of water quality parameters based on deep learning and continuous spectrum according to claim 8, characterized in that: The deep learning model is constructed, a model is selected according to the spectral wavelength characteristics corresponding to the supervised learning label, and the deep learning model is optimized by adding transfer learning, and further includes: adding a shallow convolution corresponding to the water quality parameter specific to the shared bottom layer to capture the key characteristics of the water quality parameter, and the key characteristics of the water quality parameter are not included in the characteristics of the shared layer; A dynamic weight mechanism is added to assign weight balance dimensions to different water quality parameters by designing a loss function. At the same time, the weights are automatically optimized based on uncertainty weighting. A SE module is also provided to enhance the wavelength weights corresponding to the key features of the water quality parameters through an attention mechanism.

10. The method for online monitoring of water quality parameters based on deep learning and continuous spectrum according to claim 9, characterized in that: The method of combining the isolation forest or autoencoder reconstruction error to obtain the pollutant exceeding the standard alarm specifically includes: the isolation forest identifies outliers based on the path length difference of the spectral characteristics; for the autoencoder, it is used to train AE to reconstruct the normal spectrum, calculate the reconstruction error as the anomaly score, and output the false alarm rate of the pollutant exceeding the standard alarm based on the anomaly score.

Images