A spectrum sample generation method for water COD detection

Abstract

A spectrum sample generation method for water COD detection includes the following steps: Step 1, using a variational mode decomposition algorithm to obtain spectrum characteristic components A of different types of real samples pc (λ) and the residual component A res (λ); Step 2, use the third-order Gaussian function f res (λ) uniformly expresses the residual components of all different types of samples and determines the parameter a i 、b i 、c i The adjustable range is obtained by adjusting the parameters to obtain different residual components; Step 3, use the generative adversarial network to obtain new feature components A of different types of water samples pc‑new (λ); Step 4, reconstruct the different residual components and new characteristic components by superimposing them to obtain the newly generated spectral sample A new (λ). The samples generated by the present invention can not only retain the original spectral characteristics of COD substances, but also have diversity; the regression model can achieve better prediction accuracy in COD measurement of wastewater from different factories.

Description

Technical Field

[0001] The invention relates to a spectrum sample generation method based on variational mode decomposition combined with a generative adversarial network, and is used for determining COD in water bodies. Background Art

[0002] The discharge of industrial wastewater is one of the main causes of organic pollution in water bodies. In recent years, UV-visible spectroscopy has been widely used in water quality testing, with the advantages of fast processing speed, simple detection, and no secondary pollution. Chemical oxygen demand (COD) is one of the important comprehensive indicators reflecting the degree of water pollution. At present, many methods invert the COD value by establishing a regression model between the absorption spectrum and the concentration of organic matter. However, due to the decreasing probability of water pollution incidents year by year, it is impossible to collect enough samples for modeling. And when the regression model is insufficient in training samples, the prediction results are often unsatisfactory.

[0003] At present, the commonly used spectral data generation method is to mix the measurement data obtained from laboratory experiments and real water systems in a certain proportion to obtain different types of synthetic data sets, but this method cannot accurately grasp the mixing ratio of the two solutions, and the operation of this method is relatively complicated and has poor universality. In addition, there are some sample synthesis technologies that generate samples by transforming functions or adding noise signals, but this method is prone to losing or changing the main spectral feature information of the real sample, so that there are certain defects in obtaining effective and diverse spectral samples.

[0004] In recent years, generative adversarial networks have received extensive attention in the generation of hyperspectral images and spatial spectral samples, and are often used to solve the problem of insufficient training samples. However, the spectral samples obtained by traditional generative adversarial network methods are usually very similar to the original samples, lacking diversity, and are not conducive to the establishment of prediction models. Currently, this method is rarely used directly for spectral sample generation.

[0005] Therefore, in order to establish a more accurate COD prediction model, we need to find a sample generation method to generate more effective and diverse absorption spectrum samples for modeling with less existing real data, so as to improve the prediction performance of the model. Summary of the invention

[0006] In order to overcome the above-mentioned deficiencies of the prior art and generate a spectral sample that can retain the original spectral characteristics of COD substances and has diversity, the present invention provides a spectral sample generation method for water COD detection.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] A method for generating a spectrum sample for water body COD detection, the method comprising the following steps:

[0009] Step 1: Use the t-distributed random neighbor embedding algorithm to visualize and cluster the absorption spectra of different types of actual water samples and COD standard solutions, and divide the various types of spectra into two categories for subsequent analysis;

[0010] Step 2: Based on the given absorption spectra A(λ) of different types of real water samples, the variational mode decomposition algorithm is used to decompose the spectra into characteristic components A of the signal. pc (λ) and the residual component A res (λ), where the parameters of the variational mode decomposition algorithm are pre-set: the number of modes to be decomposed, the penalty coefficient, and the convergence tolerance;

[0011] Step 3: The residual component A of all samples of different types in step 2 res (λ)Using a unified mathematical expression f res (λ) is expressed as shown in formula (1):

[0012]

[0013] Among them, a i , b i 、c i is the adjustable parameter of the Gaussian function, N is the order of the Gaussian function, and the N-order Gaussian function is used to fit the residual components of different types of water samples within the 95% confidence interval to determine the parameter a i , b i 、c i scope;

[0014] Step 4: For the characteristic components A of different types of actual samples pc (λ), using the generative adversarial network method to generate a variety of feature components A pc-new (λ), where the characteristic component A of the actual sample pc (λ) is used as the real sample of the discriminator in the generative adversarial network, and the label is 1; the random vector signal is used as the input of the generator in the generative adversarial network, and the output vector is used as the false sample of the discriminator, and the label is 0; after the generative adversarial network is trained for a set number of times, the final output result of the generator is the new feature component A we need pc-new (λ), in each training process of the generative adversarial network, the discriminator and the generator share network parameters with each other;

[0015] Step 5: randomly adjust the three parameters a of the function in step 3 within the parameter range. i , b i 、c iThe value of , get different residual components, and compare these residual components with the characteristic component A in step 4 pc-new (λ) is reconstructed by superposition; finally we get the generated sample A we need new (λ), as shown in formula (2):

[0016]

[0017] Furthermore, in step 2, the number of decomposed modes is 10, the penalty coefficient is 2000, and the convergence tolerance is 1e-7.

[0018] Furthermore, in step 3, the order N of the Gaussian function is 3, and Table 1 shows the parameter a i , b i 、c i scope,

[0019]

[0020] Table 1.

[0021] Furthermore, in step 4, the generator and discriminator of the generative adversarial network are implemented by two forward feedback neural networks, and Table 2 is the parameter setting of the neural network

[0022]

[0023] Table 2.

[0024] The technical concept of the present invention is: using the variational mode decomposition algorithm to obtain the characteristic components and residual components of the absorption spectrum to ensure that the subsequently generated samples retain the spectral characteristics of COD substances and have diversity. The function of fitting the residual classification is determined. A third-order Gaussian function is used to represent the residual components of all different water sample types. Parameter a i , b i 、c i Determination of adjustable range. The residual components of different types of water samples were fitted within the 95% confidence interval using a third-order Gaussian function to obtain the adjustable range of the parameters. Generative adversarial networks were used to generate new feature components, which not only avoided the loss of spectral features but also made the generated samples diverse.

[0025] The beneficial effects of the present invention are as follows:

[0026] 1. Based on the variational mode decomposition algorithm to obtain the characteristic components and residual components of the absorption spectrum, the present invention uses a third-order Gaussian function to represent the residual components of all different water sample types, and obtains a variety of residual components by performing random adjustments within the parameter range to ensure that the final generated samples have diversity.

[0027] 2. Based on fewer real samples, the generative adversarial network method is used to obtain new absorption spectral characteristic components and use them to reconstruct new samples, avoiding the loss of spectral characteristics of new samples. It is also beneficial to improve the accuracy of the regression model when the spectral samples are applied in COD prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the method of the present invention.

[0029] Figure 2 These are the absorption spectrum curves of wastewater samples from four factories and standard COD solution.

[0030] Figure 3 It is the visualization clustering result of the absorption spectra of different types of water samples.

[0031] Figure 4 It is the absorbance curve fitted by the residual component of different types of water samples.

[0032] Figure 5 : are the absorbance curves of the generated samples at different concentrations, (a) is the first type of generated samples, and (b) is the second type of generated samples.

[0033] Figure 6 It is the prediction result of the COD regression model of the back propagation neural network. (a) is the first type of data set, and (b) is the second type of data set. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0035] Reference Figure 1 to Figure 6 , a spectral sample generation method for water COD detection, the generated spectral samples are used to establish a COD regression model with better prediction performance. This method first uses the variational mode decomposition algorithm to obtain the spectral characteristic components A of different types of real samples pc (λ) and the residual component A res (λ). In particular, using the third-order Gaussian function f res (λ) uniformly expresses the residual components of all different types of samples, and can adjust the function parameter a i 、b i 、c i To obtain different residual components. At the same time, the generative adversarial network is used to obtain new feature components A of different types of water samples. pc-new (λ). Finally, the residual component and the new characteristic component are superimposed and reconstructed to obtain the final newly generated spectral samples. These diverse and effective spectral samples are used for COD regression modeling and predicting COD values. Figure 1

[0036] A method for generating spectral samples for water body COD detection and for establishing a COD regression model with higher prediction accuracy includes the following steps:

[0037] Step 1: Use the t-distributed stochastic neighbor embedding (t-SNE) algorithm to visualize the absorption spectra of different types of actual water samples and COD standard solutions. The absorption spectra of various types of water samples can be divided into two categories. Among them, different types of water samples are selected from four typical factory wastewaters, and the spectral data in the wavelength range of 240-500nm are selected as the research object. Figure 2 After a series of processing by the t-distributed random neighbor embedding algorithm, the samples can be displayed at different positions in the two-dimensional space. The actual spectral samples collected can finally be divided into two categories. The first and second sample sets contain 29 and 30 samples respectively. The visualization results can be referred to Figure 3 .

[0038] Step 2: Based on the classification results in step 1, the absorption spectra A(λ) of given different types of real water samples are decomposed into characteristic components A of the signal using the variational mode decomposition algorithm. pc (λ) and the residual component A res (λ).

[0039] Step 3: The residual components A of all samples of different types in step 2 are res (λ)Using a unified mathematical expression f res (λ) is expressed as shown in formula (1):

[0040]

[0041] The order N of the Gaussian function is set to 3, and then the residual components of different types of water samples are fitted within the 95% confidence interval using the third-order Gaussian function. The fitting result curve of the residual components is referenced Figure 4 . And we can get the parameter a of the Gaussian function i 、b i 、c i By randomly adjusting these three parameters within the parameter range, different residual components are obtained. i 、b i 、c i The value range is shown in Table 1 below.

[0042]

[0043] Table 1

[0044] Step 4: The characteristic components A of different types of water samples in step 2 are pc(λ) is normalized in the numerical range of [0, 1], and then the normalized result is used as the real sample input of the discriminator in the generative adversarial network, with the label 1; the random vector of the same dimension is used as the input of the generator in the generative adversarial network, and the vector obtained by the generator output is used as the false sample of the discriminator, with the label 0. Two forward feedback neural networks are used to implement the generator and discriminator of the generative adversarial network. After setting the parameters of each network, new spectral feature components are obtained in the generator after 50,000 interactive trainings. According to the spectral data classification results of step 1, the two types of spectral data sets are respectively subjected to the generative adversarial network method to obtain different types of new feature components.

[0045] The generator and discriminator of the generative adversarial network are implemented by two forward feedback neural networks. Table 2 shows the parameter settings of the neural network.

[0046]

[0047] Table 2.

[0048] Step 5: Reconstruct the new spectral characteristic components obtained in step 4 and the residual components obtained in step 3 by superposition. Finally, the reconstructed first and second spectral data sets are 1000 and 800 respectively, and the absorbance curves of the samples are referenced to Figure 5 (a) and Figure 5 (b) The absorbance curves of the generated samples show their diversity.

[0049] Step 6: Using all the above real samples and generated samples, a back propagation neural network (BPNN) is used to establish a COD regression model and predict the COD value. The spectral data with a wavelength range of 240-500nm is selected as the training data. Finally, the performance of the COD prediction model is compared between the two types of modeling data sets with and without the generated samples. The prediction results are referenced Figure 6 (a) and Figure 6 (b), correlation coefficient (R 2 ) and root mean square error (RMSE) values ​​are shown in Table 3 below.

[0050]

[0051] Table 3

[0052] from Figure 6 (a) Figure 6(b) and the prediction results in Table 3 show that when building the COD prediction model, after adding more effective generated data, the correlation coefficient of the COD regression model is higher, and the value of the root mean square error is significantly reduced. The results show that the samples generated based on variational mode decomposition and generative adversarial network methods can be effectively used in COD prediction and can solve the problem of insufficient training data.

[0053] The contents described in the embodiments of this specification are merely an enumeration of implementation forms of the inventive concept and are not intended to limit the present invention. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for generating spectral samples for water COD detection, characterized in that: The method comprises the following steps: Step 1: Use the t-distributed random neighbor embedding algorithm to visualize and cluster the absorption spectra of different types of actual water samples and COD standard solutions, and divide the various types of spectra into two categories for subsequent analysis; Step 2: Based on the given absorption spectra of different types of real water samples A raw (λ), the spectrum is decomposed into the characteristic components A of the signal using the variational mode decomposition algorithm pc (λ) and the residual component A res (λ), where the parameters of the variational mode decomposition algorithm are pre-set: the number of modes to be decomposed, the penalty coefficient, and the convergence tolerance; Step 3: The residual component A of all samples of different types in step 2 res (λ)Using a unified mathematical expression f res (λ) is expressed as shown in formula (1): Among them, a i , b i 、c i is the adjustable parameter of the Gaussian function, N is the order of the Gaussian function, and the N-order Gaussian function is used to fit the residual components of different types of water samples within the 95% confidence interval to determine the parameter a i , b i 、c i scope; Step 4: For the characteristic components A of different types of actual samples pc (λ), using the generative adversarial network method to generate a variety of feature components A pc-new (λ), where the characteristic component A of the actual sample pc (λ) is used as the real sample of the discriminator in the generative adversarial network, and the label is 1; the random vector signal is used as the input of the generator in the generative adversarial network, and the output vector is used as the false sample of the discriminator, and the label is 0; after the generative adversarial network is trained for a set number of times, the final output result of the generator is the new feature component A we need pc-new (λ), in each training process of the generative adversarial network, the discriminator and the generator share network parameters with each other; Step 5: randomly adjust the three parameters a of the function in step 3 within the parameter range. i , b i 、c i The value of , get different residual components, and compare these residual components with the characteristic component A in step 4 pc-new (λ) is reconstructed by superposition; finally the required generated sample A is obtained new (λ), as shown in formula (2):

2. A method for generating spectral samples for water COD detection according to claim 1, characterized in that: In step 2, the number of decomposed modes is 10, the penalty coefficient is 2000, and the convergence tolerance is 1e-7.

Images