Modeling method, system and medium for small sample total nitrogen ultraviolet spectrum detection based on weak supervision

Abstract

The present invention discloses a method, system and medium for modeling of ultraviolet spectrum detection of total nitrogen in a small sample based on weak supervision. The method comprises the following steps: for a training set water sample, obtaining a temporally continuous total nitrogen concentration reference value and ultraviolet spectrum data in a digestion process; based on the ultraviolet spectrum data, using spectrum clustering to form a spatial and temporal relationship network between the training set water samples, and then establishing a spatiotemporal relationship network; based on the total nitrogen concentration parameter value, ultraviolet spectrum data and the spatiotemporal relationship network, constructing a machine learning target loss function based on weak supervision, and optimizing and solving to obtain a weight matrix; using the weight matrix to correct the total nitrogen concentration reference value of the training set water sample, and using the obtained correction value as the total nitrogen concentration label value of the training set water sample; using the ultraviolet spectrum data and the total nitrogen concentration label value of the training set water sample as input and output data, respectively, to train a total nitrogen spectrum rapid detection model. The present invention can correct the total nitrogen concentration reference value, thereby establishing an accurate total nitrogen spectrum detection model.

Description

Technical Field

[0001] The invention belongs to the field of automatic detection, and relates to a small sample total nitrogen ultraviolet spectrum detection modeling method, system and medium based on ultraviolet spectrum analysis and weak supervision. Background Art

[0002] As the main indicator for evaluating water pollution, the total nitrogen concentration of water bodies can directly reflect the degree of organic pollution. Its rapid and reliable online detection can effectively prevent a series of environmental pollution disasters such as algal blooms and eutrophication of water bodies. Based on the fingerprint absorption characteristics of the final product of total nitrogen oxidation digestion in the ultraviolet spectrum, we can quickly and reliably detect the total nitrogen concentration of water bodies by in-situ detecting the dynamic changes of the ultraviolet spectrum during the oxidation digestion process of the water sample to be tested. Considering that the total nitrogen components of different water bodies are different, the ultraviolet spectrum change characteristics of different water bodies during the digestion process are also different. It is necessary to establish different total nitrogen spectrum detection models according to the water quality characteristics of different water bodies. However, manual offline calibration of the total nitrogen sample concentration value is time-consuming and laborious, and it is often difficult to obtain sufficient calibration training set samples in the actual detection process, which greatly limits the actual application effect of total nitrogen detection.

[0003] Weakly supervised machine learning algorithm is a new machine learning method that has emerged in recent years. Its importance can be divided into three categories: incomplete supervision, inexact supervision and inaccurate supervision. Incomplete supervision means that only part of the training data is labeled, and a large amount of data is not labeled; inexact supervision mainly refers to the training samples in the classification problem that only have coarse-grained labels; the definition of inaccurate supervision is that the sample labels in a given training set are not always true values. For total nitrogen detection, since the reference value of its training samples is provided by the online detection instrument, its concentration label is often far from the true value, which is a kind of inaccurate supervision. Summary of the invention

[0004] In order to overcome the problems that the true value of total nitrogen concentration of training samples is difficult to obtain and the training samples of spectral detection models are insufficient in the actual detection process, the present invention proposes a new small-sample total nitrogen ultraviolet spectral detection modeling method based on weak supervision, which can correct the large-deviation total nitrogen concentration reference value, thereby establishing an accurate total nitrogen spectral detection model, and thus overcoming the problem that the training set sample labels in traditional modeling methods need to be manually calibrated offline.

[0005] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0006] A weakly supervised small sample total nitrogen ultraviolet spectrum detection modeling method, comprising:

[0007] Step 1: For the training set water samples, a total nitrogen automatic calibrator is used to obtain temporally continuous total nitrogen concentration reference values, and a UV spectrometer is used to detect UV spectrum data during the digestion process;

[0008] Step 2, based on the ultraviolet spectrum data of the training set water samples during digestion, spectral clustering is used to construct a spatial relationship network and a temporal relationship network between the training set water samples, and then a spatiotemporal relationship network of the training set water samples is established;

[0009] Step 3, based on the total nitrogen concentration parameter value, ultraviolet spectrum data and spatiotemporal relationship network of the training set water samples, a machine learning target loss function based on weak supervision is constructed, and the weight matrix of the ultraviolet spectrum data of the training set water samples during the digestion process is obtained by optimization;

[0010] Step 4, using the weight matrix obtained in step 3 to correct the reference value of the total nitrogen concentration of the training set water sample, and using the obtained corrected value as the label value of the total nitrogen concentration of the training set water sample;

[0011] Step 5: Use the UV spectral data and total nitrogen concentration label values ​​of the training set water samples during the digestion process as input and output data, respectively, to train the total nitrogen spectral rapid detection model.

[0012] Furthermore, the detection error of the total nitrogen automatic calibrator should be less than 10%.

[0013] Furthermore, the total nitrogen concentration reference value and ultraviolet spectrum data of the training set water samples were obtained by the following steps:

[0014] Step 1.1: Add 1 ml of the sample solution to be tested and 4 ml of alkaline potassium persulfate oxidant into the oxidation digestion reaction cup, and use the obtained oxidation digestion mixed solution as the training set water sample;

[0015] The alkaline potassium persulfate solution is prepared by mixing 1g / l NaOH and 1g / l K2S2O8 in a ratio of 1:1;

[0016] Step 1.2: With t as the time interval, use the total nitrogen automatic calibrator to detect the reference value of the total nitrogen concentration in the training set water samples, use the ultraviolet spectrometer to detect the ultraviolet spectrum changes of the training set water samples, and use the heating resistance wire surrounding the reaction cup to heat the training set water samples to 120°C;

[0017] The UV spectrum measurement range is 200nm-300nm, and the time interval t is 1s;

[0018] Step 1.3: After the training set water sample is heated to 120°C, maintain the temperature for 5 minutes, and continuously measure the reference value of total nitrogen concentration and ultraviolet spectrum changes in the training set water sample at time intervals of t;

[0019] Step 1.4: Use a cooling fan to reduce the temperature of the training set water samples to 50°C, ending the entire UV spectrum data collection process.

[0020] Furthermore, the specific process of step 2 is:

[0021] Step 2.1: Calculate the Euclidean distance of the UV spectra between different water samples in the training set, and use this to construct the spatial distance relationship matrix D of the training set s ;

[0022] The distance relationship matrix D s Every element of It can be calculated as follows:

[0023]

[0024] x i is the UV spectrum data of the i-th water sample in the training set during the digestion process;

[0025] Step 2.2: Based on spectral clustering, a similarity matrix A between different water samples in the training set is constructed, so that the UV spectrum data of two water samples with a larger Euclidean distance have a smaller similarity, and the UV spectrum data of each water sample is only associated with τ other water samples;

[0026] Each element a of the similarity matrix A is ij By solving the following problem:

[0027]

[0028]

[0029] γ is a regularization parameter; 1 is a unit vector; a i is the i-th row vector of the similarity matrix A; τ is a hyperparameter; n is the number of samples in the training set;

[0030] Solving this optimization problem according to Lagrange number multiplication yields:

[0031]

[0032] Step 2.3: Use the obtained similarity matrix A to construct the Laplace spatial relationship network L between different water samples in the training set s :

[0033] L s =Diag-A (4)

[0034] Where Diag is a diagonal matrix, and its i-th diagonal element can be calculated by the following formula:

[0035] dii =∑ j a ij (5)

[0036] Step 2.4: Calculate the time distance matrix D between different water samples according to the detection time of the water samples t , calculate the Laplace time relationship network L between different water samples in the training set according to the method of steps 2.2-2.3 T , and then obtain the final spatiotemporal relationship network L of the training set water samples;

[0037] Where D t Each element of It can be calculated by the following formula:

[0038]

[0039] Where, T i and T j Respectively represent the acquisition time of the UV spectrum data of the i-th and j-th water samples in the training set, and the elements Represents x i With x j The time distance between

[0040] The spatiotemporal relationship network L is calculated by the following formula:

[0041] L=L S +μL T (7)

[0042] Where μ is a hyperparameter used to adjust the importance between the temporal relationship network and the spatial relationship network.

[0043] Furthermore, the objective loss function of machine learning based on weak supervision is constructed as follows:

[0044]

[0045] Where X is the matrix composed of the ultraviolet spectral data of all water samples in the training set during the digestion process, and the first column is 1 sample; Y is the column vector composed of the total nitrogen reference values ​​of all water samples in the training set; W is the weight matrix of machine learning, that is, the weight matrix of the ultraviolet spectral data of water samples in the training set during the digestion process, and β and λ are hyperparameters.

[0046] Furthermore, the Lagrange number multiplication is used to solve the target loss function to obtain the weight matrix:

[0047] W=(XX T +βXLX T ) -1 XY (9).

[0048] A computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of any of the above-mentioned weakly supervised small sample total nitrogen ultraviolet spectrum detection modeling methods are implemented.

[0049] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements any weakly supervised small sample total nitrogen ultraviolet spectrum detection modeling method.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] In view of the problems of large demand for training samples and time-consuming and laborious manual offline calibration of sample labels in traditional total nitrogen detection modeling methods, the present invention adopts a weakly supervised machine learning method, based on the inaccurate total nitrogen concentration reference value provided by the existing online detection instrument, and corrects the total nitrogen reference value of different training samples online through the spatial similarity of ultraviolet spectrum changes in the digestion process of different samples and the temporal similarity in the continuous sampling process, eliminating the need for manual offline acquisition of total nitrogen concentration values ​​of training samples, thereby getting rid of the limitation of manual offline calibration of total nitrogen concentration samples. The corrected total nitrogen value can be used to train the total nitrogen spectrum rapid detection model to realize the machine intelligent calibration of the total nitrogen online detector. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of the modeling method for small sample total nitrogen ultraviolet spectrum detection based on weak supervision;

[0053] Figure 2 It is a schematic diagram of the changes in the UV spectrum of a water sample during the oxidation digestion process;

[0054] Figure 3 It is a spatiotemporal relationship network constructed based on the ultraviolet spectrum change diagram during oxidation digestion;

[0055] Figure 4 It is a comparison chart between the reference value of total nitrogen concentration, the sample correction value after algorithm correction and the true value of training samples. DETAILED DESCRIPTION

[0056] The following is a detailed description of an embodiment of the present invention. This embodiment is based on the technical solution of the present invention, and provides a detailed implementation method and a specific operation process to further explain the technical solution of the present invention.

[0057] A weakly supervised small sample total nitrogen ultraviolet spectroscopy detection modeling method, reference Figure 1 As shown, the following steps are included:

[0058] Step 1: For the training set water samples, a total nitrogen automatic calibrator is used to obtain temporally continuous total nitrogen concentration reference values, and a UV spectrometer is used to detect UV spectrum data during the digestion process;

[0059] The total nitrogen automatic calibration instrument in this embodiment is a total nitrogen online detector commonly used in the market, which is required to ensure that the relative total nitrogen concentration detection error is less than 10%.

[0060] The total nitrogen concentration reference value and ultraviolet spectrum data of the training set water samples are obtained by the following steps:

[0061] Step 1.1: Add 1 ml of the sample solution to be tested and 4 ml of alkaline potassium persulfate oxidant into the oxidation digestion reaction cup, and use the obtained oxidation digestion mixed solution as the training set water sample;

[0062] The alkaline potassium persulfate solution is prepared by mixing 1g / l NaOH and 1g / l K2S2O8 in a ratio of 1:1;

[0063] Step 1.2: With t as the time interval, use the total nitrogen automatic calibrator to detect the reference value of the total nitrogen concentration in the training set water samples, use the ultraviolet spectrometer to detect the ultraviolet spectrum changes of the training set water samples, and use the heating resistance wire surrounding the reaction cup to heat the training set water samples to 120°C;

[0064] The UV spectrum measurement range is 200nm-300nm, and the time interval t is 1s;

[0065] Step 1.3: After the training set water sample is heated to 120°C, maintain the temperature for 5 minutes, and continuously measure the reference value of total nitrogen concentration and ultraviolet spectrum changes in the training set water sample at time intervals of t;

[0066] Step 1.4: Use a cooling fan to reduce the temperature of the training set water samples to 50°C, ending the entire UV spectrum data collection process.

[0067] Step 2: Based on the ultraviolet spectrum data of the training set water samples during digestion, spectral clustering is used to construct the spatial relationship network and temporal relationship network between the training set water samples, and then the spatiotemporal relationship network of the training set water samples is established. Specifically, the following steps are included:

[0068] Step 2.1: Calculate the Euclidean distance of the UV spectra between different water samples in the training set, and use this to construct the spatial distance relationship matrix D of the training set s ;

[0069] The distance relationship matrix D s Every element of It can be calculated as follows:

[0070]

[0071] x i is the UV spectrum data of the i-th water sample in the training set during the digestion process;

[0072] Step 2.2: Based on spectral clustering, a similarity matrix A between different water samples in the training set is constructed, so that the UV spectrum data of two water samples with a larger Euclidean distance have a smaller similarity, and the UV spectrum data of each water sample is only associated with τ other water samples;

[0073] Each element a of the similarity matrix A is ij By solving the following problem:

[0074]

[0075]

[0076] γ is a regularization parameter; 1 is a unit vector; a i is the i-th row vector of the similarity matrix A; τ is a hyperparameter that can be selected by yourself and the range is generally 5-20 depending on the specific data set;; n is the number of samples in the training set;

[0077] Solving this optimization problem according to Lagrange number multiplication yields:

[0078]

[0079] Step 2.3: Use the obtained similarity matrix A to construct the Laplace spatial relationship network L between different water samples in the training set s :

[0080] L s =Diag-A (4)

[0081] Where Diag is a diagonal matrix, and its i-th diagonal element can be calculated by the following formula:

[0082] d ii =∑ j a ij (5)

[0083] Step 2.4: Calculate the time distance matrix D between different water samples according to the detection time of the water samples t , calculate the Laplace time relationship network L between different water samples in the training set according to the method of steps 2.2-2.3 T , and then obtain the final spatiotemporal relationship network L of the training set water samples;

[0084] Where D t Each element of It can be calculated by the following formula:

[0085]

[0086] Where, T i and T j Respectively represent the acquisition time of the UV spectrum data of the i-th and j-th water samples in the training set, and the elements Represents x i With x j The time distance between

[0087] The spatiotemporal relationship network L is calculated by the following formula:

[0088] L=L S +μL T (7)

[0089] Where μ is a hyperparameter used to adjust the importance between the temporal relationship network and the spatial relationship network.

[0090] Step 3: Based on the total nitrogen concentration parameter value, ultraviolet spectral data and spatiotemporal relationship network of the training set water samples, a machine learning target loss function based on weak supervision is constructed, and the weight matrix of the ultraviolet spectral data of the training set water samples during the digestion process is obtained by optimization.

[0091] Among them, the constructed machine learning objective loss function based on weak supervision is specifically:

[0092]

[0093] Where X is the matrix composed of the ultraviolet spectral data of all water samples in the training set during the digestion process, and the first column is 1 sample; Y is the column vector composed of the total nitrogen reference values ​​of all water samples in the training set; W is the weight matrix of machine learning, that is, the weight matrix of the ultraviolet spectral data of water samples in the training set during the digestion process, and β and λ are hyperparameters.

[0094] Then the Lagrange number multiplication is used to solve the target loss function to obtain the weight matrix:

[0095] W=(XX T +βXLX T ) -1 XY (9).

[0096] Step 4: Use the weight matrix W obtained in step 3 to correct the total nitrogen concentration reference value Y of the training set water samples, and use the corrected value As the total nitrogen concentration label value of the water samples in the training set.

[0097] Step 5: Using the ultraviolet spectrum data X and the total nitrogen concentration label value Y of the training set water samples during the digestion process as input and output data, respectively, to train the total nitrogen spectrum rapid detection model.

[0098] The trained total nitrogen spectrum rapid detection model can be used to output the total nitrogen concentration of the solution to be tested when the ultraviolet spectrum data of the solution to be tested with unknown total nitrogen concentration is input.

[0099] Example:

[0100] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0101] Take the weak supervision modeling of rapid online detection of total nitrogen concentration in a natural lake as an example. The first step is to use the existing online total nitrogen detector to provide a total nitrogen concentration reference value for each training set sample; the second step is to use a micro-ultraviolet spectrometer to in-situ detect the changes in the ultraviolet spectra of different training samples during the oxidation digestion process to obtain a three-dimensional oxidation digestion process spectrum diagram ( Figure 2 ); The third step is to construct a spatiotemporal similarity relationship network between different training samples based on the three-dimensional oxidation digestion process spectrum ( Figure 3 ); The fourth step is to correct the total nitrogen concentration calibration value provided by the total nitrogen online instrument based on weak supervision ( Figure 4 The total nitrogen concentration value of the training sample was corrected and the total nitrogen spectrum rapid detection model was established. Compared with the traditional total nitrogen online detection modeling method, this method overcomes the need for manual offline detection and realizes machine intelligent calibration. The obtained modeling detection results are shown in Table 1. It can be seen from Table 1 that this method can effectively reduce the decrease in modeling accuracy caused by inaccurate training set sample labels.

[0102] Table 1 Comparison of detection performance between total nitrogen spectral modeling method based on weak supervision and traditional modeling method

[0103]

[0104] The above embodiments are preferred embodiments of the present application. Ordinary technicians in this field can also make various changes or improvements on this basis. Without departing from the overall concept of the present application, these changes or improvements should fall within the scope of protection required by the present application.

Claims

1. A weakly supervised small sample total nitrogen ultraviolet spectrum detection modeling method, characterized in that: include: Step 1: For the training set water samples, a total nitrogen automatic calibrator is used to obtain temporally continuous total nitrogen concentration reference values, and a UV spectrometer is used to detect UV spectrum data during the digestion process; Step 2, based on the ultraviolet spectrum data of the training set water samples during digestion, spectral clustering is used to construct a spatial relationship network and a temporal relationship network between the training set water samples, and then a spatiotemporal relationship network of the training set water samples is established; The specific process of step 2 is: Step 2.1: Calculate the Euclidean distance of the UV spectra between different water samples in the training set, and use this to construct the spatial distance relationship matrix D of the training set s ; The distance relationship matrix D s Every element of It can be calculated as follows: x i is the UV spectrum data of the i-th water sample in the training set during the digestion process; Step 2.2: Based on spectral clustering, a similarity matrix A between different water samples in the training set is constructed, so that the UV spectrum data of two water samples with a larger Euclidean distance have a smaller similarity, and the UV spectrum data of each water sample is only associated with τ other water samples; Each element a of the similarity matrix A is ij By solving the following problem: γ is a regularization parameter; 1 is a unit vector; a i is the i-th row vector of the similarity matrix A; τ is a hyperparameter; n is the number of samples in the training set; Solving this optimization problem according to Lagrange number multiplication yields: Step 2.3: Use the obtained similarity matrix A to construct the Laplace spatial relationship network L between different water samples in the training set s : L s =Diag-A (4) Where Diag is a diagonal matrix, and its i-th diagonal element can be calculated by the following formula: d ii =∑ j a ij (5) Step 2.4: Calculate the time distance matrix D between different water samples according to the detection time of the water samples t , calculate the Laplace time relationship network L between different water samples in the training set according to the method of steps 2.2-2.3 T , and then obtain the final spatiotemporal relationship network L of the training set water samples; Where D t Each element of It can be calculated by the following formula: Where, T i and T j Respectively represent the acquisition time of the UV spectrum data of the i-th and j-th water samples in the training set, and the elements Represents x i With x j The time distance between The spatiotemporal relationship network L is calculated by the following formula: L=L S +μL T (7) In the formula, μ is a hyperparameter used to adjust the importance between the temporal relationship network and the spatial relationship network; Step 3, based on the total nitrogen concentration parameter value, ultraviolet spectrum data and spatiotemporal relationship network of the training set water samples, a machine learning target loss function based on weak supervision is constructed, and the weight matrix of the ultraviolet spectrum data of the training set water samples during the digestion process is obtained by optimization; Step 4, using the weight matrix obtained in step 3 to correct the reference value of the total nitrogen concentration of the training set water sample, and using the obtained corrected value as the label value of the total nitrogen concentration of the training set water sample; Step 5: Use the UV spectral data and total nitrogen concentration label values ​​of the training set water samples during the digestion process as input and output data, respectively, to train the total nitrogen spectral rapid detection model.

2. The method according to claim 1, characterized in that The detection error of the total nitrogen automatic calibrator should be less than 10%.

3. The method according to claim 1, characterized in that The total nitrogen concentration reference value and UV spectrum data of the training set water samples were obtained by following the steps below: Step 1.1: Add 1 ml of the sample solution to be tested and 4 ml of alkaline potassium persulfate oxidant into the oxidation digestion reaction cup, and use the obtained oxidation digestion mixed solution as the training set water sample; The alkaline potassium persulfate solution is prepared by mixing 1g / l NaOH and 1g / l K2S2O8 in a ratio of 1:1; Step 1.2: With t as the time interval, use the total nitrogen automatic calibrator to detect the reference value of the total nitrogen concentration in the training set water samples, use the ultraviolet spectrometer to detect the ultraviolet spectrum changes of the training set water samples, and use the heating resistance wire surrounding the reaction cup to heat the training set water samples to 120°C; The UV spectrum measurement range is 200nm-300nm, and the time interval t is 1s; Step 1.3: After the training set water sample is heated to 120°C, maintain the temperature for 5 minutes, and continuously measure the reference value of total nitrogen concentration and ultraviolet spectrum changes in the training set water sample at time intervals of t; Step 1.4: Use a cooling fan to reduce the temperature of the training set water samples to 50°C, ending the entire UV spectrum data collection process.

4. The method according to claim 1, characterized in that The constructed objective loss function of machine learning based on weak supervision is as follows: Where X is the matrix composed of the ultraviolet spectral data of all water samples in the training set during the digestion process, and the first column is 1 sample; Y is the column vector composed of the total nitrogen reference values ​​of all water samples in the training set; W is the weight matrix of machine learning, that is, the weight matrix of the ultraviolet spectral data of water samples in the training set during the digestion process, and β and λ are hyperparameters.

5. The method according to claim 4, characterized in that The Lagrange number multiplication is used to solve the target loss function to obtain the weight matrix: W=(XX T +βXLX T ) -1 XY (9)。 6. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for modeling the detection of total nitrogen by ultraviolet spectrum of small samples based on weak supervision as described in any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images