Ammonia nitrogen rapid detection method and equipment based on gradient boosting ensemble learning algorithm and three-dimensional fluorescence

Abstract

This invention discloses a rapid detection method and device for ammonia nitrogen in water quality based on gradient boosting ensemble learning algorithm and three-dimensional fluorescence. The method includes: (1) collecting three-dimensional fluorescence spectral data and ammonia nitrogen concentration data of water samples, preprocessing them, and constructing a dataset, wherein the dataset includes a training dataset and a test dataset; (2) constructing a gradient boosting regression tree model and training it using the training dataset to generate an ammonia nitrogen prediction model; (3) collecting three-dimensional fluorescence spectral data of the water sample to be tested, preprocessing it, and inputting it into the ammonia nitrogen prediction model to predict the ammonia nitrogen concentration of the water sample to be tested. This invention makes full use of all three-dimensional fluorescence spectral matrix data, reflecting the true situation of the data to the maximum extent without omitting any information, thereby accurately predicting the ammonia nitrogen concentration.

Description

Technical Field

[0001] This invention relates to the field of water quality testing technology, and in particular to a rapid method and device for detecting ammonia nitrogen in water quality based on gradient boosting ensemble learning algorithm and three-dimensional fluorescence. Background Technology

[0002] Three-dimensional fluorescence spectroscopy is a technique for analyzing fluorescent substances in water, which can reflect the organic matter content and structural characteristics of water quality.

[0003] Three-dimensional fluorescence water quality monitoring has advantages such as high sensitivity, good selectivity, fast speed, and no need for pretreatment. It can provide information that ordinary fluorescence spectroscopy cannot provide, such as the source, type, structure, and changes of the pollutants being measured. Three-dimensional fluorescence water quality monitoring has wide applications in various water environments, including domestic sewage, industrial wastewater, rainwater, surface water, and groundwater. It can monitor water quality changes in real time, assess the degree of water pollution, and guide water environment management and protection.

[0004] Ammonia nitrogen is a common water pollution indicator that has adverse effects on aquatic ecosystems and human health. Rapid prediction of ammonia nitrogen concentration in water using three-dimensional fluorescence spectroscopy enables online monitoring and real-time early warning of water quality, allowing for timely detection of water quality anomalies and improving water quality management efficiency and water environment protection levels. It also avoids the complex sample pretreatment and time-consuming instrument operation required by traditional chemical analysis methods, reducing the cost of water quality testing, human error, and secondary pollution problems.

[0005] Currently, three-dimensional fluorescence technology is mainly used for source tracing in water quality testing, but there is no effective method to directly extract ammonia nitrogen characteristics from three-dimensional fluorescence intensity. Currently, dimensionality reduction methods such as principal component analysis (PCA) and parallel factor analysis (PARAFAC) are commonly used for 3D fluorescence spectroscopy preprocessing to extract useful information from the 3D fluorescence spectrum while removing noise and redundant data. However, these dimensionality reduction methods result in the loss of some 3D fluorescence features, which is detrimental to the extraction of ammonia nitrogen concentration characteristics and the establishment of ammonia nitrogen concentration prediction models.

[0006] For example, Chinese patent document CN115236048A discloses a method for monitoring ammonia nitrogen concentration in water based on three-dimensional fluorescence spectroscopy. It uses parallel factor analysis to analyze the three-dimensional fluorescence spectral matrix data, first obtaining the organic component fraction, and then establishing a correlation model between the fluorescence intensity of the organic components and the ammonia nitrogen concentration. However, this method fails to directly extract the ammonia nitrogen characteristics from the three-dimensional fluorescence, resulting in low accuracy of the final ammonia nitrogen concentration prediction model.

[0007] Due to the large volume of 3D fluorescence data, there is currently no efficient method to extract the ammonia nitrogen concentration features from the 3D fluorescence intensity data for ammonia nitrogen concentration prediction. This invention combines the gradient boosting regression tree algorithm from machine learning, using all 3D fluorescence intensities as independent features for model training. This allows for the acquisition of key features of ammonia nitrogen concentration from the 3D fluorescence intensity, establishing a prediction model to achieve the goal of predicting ammonia nitrogen concentration in water quality. Summary of the Invention

[0008] This invention provides a rapid method for detecting ammonia nitrogen in water quality based on gradient boosting ensemble learning algorithm and three-dimensional fluorescence. It makes full use of all three-dimensional fluorescence spectral matrix data, and reflects the true situation of the data to the greatest extent without missing any information, thereby accurately predicting the ammonia nitrogen concentration.

[0009] The technical solution of the present invention is as follows:

[0010] A rapid method for detecting ammonia nitrogen in water quality based on gradient boosting ensemble learning algorithm and three-dimensional fluorescence includes:

[0011] (1) Collect three-dimensional fluorescence spectrum data and ammonia nitrogen concentration data of water samples, preprocess them and construct a dataset, which includes a training dataset and a test dataset;

[0012] (2) Construct a gradient boosting regression tree (GBRT) model and train it using the training dataset to generate an ammonia nitrogen prediction model;

[0013] (3) Collect the three-dimensional fluorescence spectrum data of the water sample to be tested, preprocess it, and input it into the ammonia nitrogen prediction model to predict the ammonia nitrogen concentration of the water sample to be tested.

[0014] This invention combines the gradient boosting regression tree algorithm from machine learning to train a model using all three-dimensional fluorescence intensities of a water sample as independent features. This allows for the acquisition of key features related to ammonia nitrogen concentration within the three-dimensional fluorescence intensity, enabling the establishment of an ammonia nitrogen prediction model and achieving the goal of predicting ammonia nitrogen concentration in water quality. This invention does not perform dimensionality reduction on the three-dimensional fluorescence spectral matrix data, thus preserving the three-dimensional fluorescence features and resulting in a more accurate ammonia nitrogen prediction model.

[0015] When collecting the three-dimensional fluorescence spectral matrix data of water samples, the detection conditions were as follows: excitation wavelength (Ex) 220-450 nm, scanning interval 5 nm, slit width 5 nm; emission wavelength (Em) 260-600 nm, scanning interval 1 nm, slit width 5 nm; and scanning speed 2400 nm / min.

[0016] The ammonia nitrogen concentration data of the water samples were detected using Nessler's reagent method.

[0017] To eliminate the dimensional differences between different features and make the data more suitable for model training, it is necessary to preprocess the three-dimensional fluorescence spectrum data and ammonia nitrogen concentration data of the water samples.

[0018] Preferably, Z-score normalization is used to perform a linear transformation on the original matrix of the three-dimensional fluorescence spectral data and the ammonia nitrogen concentration data of the water sample to obtain standardized three-dimensional fluorescence spectral data and ammonia nitrogen concentration data.

[0019] The formula for linearly transforming the original matrix of the three-dimensional fluorescence spectral data and the ammonia nitrogen concentration data of the water sample using Z-score normalization is as follows:

[0020]

[0021] In the formula, x is the standardized data value, X is the original data value, and X... mean X is the average of the original data values. std This represents the standard deviation of the original data values.

[0022] In step (2), the gradient boosting regression tree model is constructed and trained using the gradient boosting regression tree algorithm in the scikit-learn library.

[0023] Furthermore, step (2) includes:

[0024] (2-1) Define the training set D = {(x1,y1), (x2,y2), ..., (x...} n ,y n )}, where n is the number of training samples in the training set, x i (i = 1, 2…n) represents the three-dimensional fluorescence spectral data of the training samples, y i (i = 1, 2, ..., n) represents the ammonia nitrogen concentration data of the training samples, M represents the maximum number of iterations, the loss function is L(y, F(x)), and F(x) represents the output regression tree model;

[0025] (2-2) Initialize the weak learner:

[0026]

[0027] Among them, F ( (x) is a weak learner, and L is the loss function;

[0028] (2-3) Minimize the loss function:

[0029]

[0030] h my is the predicted value of the j-th leaf node in the m-th iteration (1,2,…,M), where j is the number of leaf nodes in the m-th regression tree, L is the loss function, and y i F is the true value of the i-th sample. m-1 (x i h(x) is the predicted value of the model for the i-th sample after the (m-1)-th iteration. * The predicted value of the j-th leaf node;

[0031] (2-4) Update the model:

[0032] F m (x)=F m-, (x)+γ·h m (x)

[0033] F m (x) represents the model prediction value after the m-th iteration, F m-, (x) represents the model prediction value after the (m-1)th iteration, γ represents the learning rate, and h m (x) represents the predicted value of the regression tree obtained in the m-th iteration of training;

[0034] (2-5) Repeat steps (2-3)-(2-4) until the maximum number of iterations is reached, and sum the models to obtain the final ammonia nitrogen prediction model:

[0035]

[0036] Preferably, mean squared error (MSE) is used as the loss function.

[0037] Preferably, when training the gradient boosting regression tree model, the learning rate γ is set to 0.1, the number of trees is set to 50-100, the maximum depth is 2-4, the minimum number of samples required for node splitting is 2, the minimum number of samples required for leaf nodes is 1, and the sample sampling ratio for each tree is 1.0. Based on the same inventive concept, this invention also provides a rapid water quality ammonia nitrogen detection device based on a gradient boosting ensemble learning algorithm and three-dimensional fluorescence. The rapid water quality ammonia nitrogen detection device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory stores an ammonia nitrogen prediction model, and the processor executes the computer program to perform the following steps:

[0038] The three-dimensional fluorescence spectral data of the water sample to be tested is preprocessed and input into the ammonia nitrogen prediction model to predict the ammonia nitrogen concentration of the water sample and output the prediction results.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0040] The method of this invention does not require the use of traditional dimensionality reduction techniques such as principal component analysis (PCA) and parallel factor analysis (PARAFAC) for the analysis of three-dimensional fluorescence spectral data. By treating all three-dimensional fluorescence intensities as independent features and training a gradient boosting regression tree model, it efficiently extracts key spectral features closely related to ammonia nitrogen concentration in water quality, establishes a prediction model, and thus quickly and accurately achieves the goal of predicting ammonia nitrogen concentration in water quality. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating the rapid detection method for ammonia nitrogen in water quality based on gradient boosting ensemble learning algorithm and three-dimensional fluorescence according to the present invention.

[0042] Figure 2 A schematic diagram of three-dimensional fluorescence data visualization;

[0043] Figure 3 This is an example of ammonia nitrogen concentration features in the three-dimensional fluorescence data obtained from the model. Detailed Implementation

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not limit it in any way.

[0045] Example 1

[0046] like Figure 1 As shown, the present invention provides a rapid water quality ammonia nitrogen detection method based on gradient boosting ensemble learning algorithm and three-dimensional fluorescence, comprising:

[0047] (1) Data Acquisition and Analysis: Ammonia nitrogen and three-dimensional fluorescence data were analyzed from the collected surface water and sewage samples from river outlets. Three-dimensional fluorescence spectroscopy detection conditions: excitation wavelength (Ex) 220–450 nm, scanning interval 5 nm, slit width 5 nm; emission wavelength (Em) 260–600 nm, sampling interval 1 nm, slit width 5 nm; scanning speed 2400 nm / min.

[0048] Nessler's reagent method was used for ammonia nitrogen detection.

[0049] (2) Preprocessed data:

[0050] Z-score standardization (calculation formula below) was adopted. By performing a linear transformation on the original matrix of water quality three-dimensional fluorescence spectrum and ammonia nitrogen concentration data, the mean of the data was 0 and the standard deviation was 1, thereby eliminating the dimensional differences between different features and making the data more suitable for model training.

[0051]

[0052] In the formula, x is the standardized data value, X is the original data value, and X... mean X is the average of the original data values. std This represents the standard deviation of the original data values.

[0053] (3) Data partitioning:

[0054] Randomly select a portion of the standardized data to create training and test sets for model training.

[0055] (4) Model training:

[0056] The gradient boosting tree algorithm and the random forest ensemble algorithm from the scikit-learn library were used for model training respectively. The prediction results of the different algorithms were compared and evaluated.

[0057] 1) Gradient boosting regression tree model training

[0058] Gradient boosting regression trees gradually reduce residuals by sequentially training a series of decision trees, thereby progressively improving the model's predictive ability. By iteratively training the decision trees and continuously optimizing the loss function, a powerful ensemble model is ultimately obtained.

[0059] First, a simple model (e.g., the average value) is initialized to fit the target value of the training data as the initial prediction. The residuals of the current model are calculated, and a new decision tree is fitted to predict these residuals, ensuring that the current model combined with the new tree's prediction better approximates the actual value. Then, gradient descent is used to minimize the loss function and update the model parameters to reduce the residuals. Finally, the predictions from each tree are summed to obtain the final model prediction.

[0060] The specific steps are as follows:

[0061] For the training set D = {(x1,y1), (x2,y2), ..., (x...} n ,y n )}, where n is the number of training samples, x i (i = 1, 2…n) represents the three-dimensional fluorescence spectral data of the training samples, y i (i = 1, 2, ..., n) represents the ammonia nitrogen concentration of the training samples. The maximum number of iterations is M, the mean squared error (MSE) is used as the loss function L(y, F(x)), and the output is the regression tree model F(x).

[0062] a. Initialize the weak learner

[0063]

[0064] Among them, F ( (x) is a weak learner, and L is the loss function.

[0065] b. Minimize the loss function

[0066] For iteration rounds m = 1, 2, ..., M

[0067]

[0068] h m y is the predicted value of the j-th leaf node in the m-th iteration, where j is the number of leaf nodes in the m-th decision tree, L is the loss function, and y i F is the true value of the i-th sample. m-1 (x i h(x) is the predicted value of the model for the i-th sample after the (m-1)-th iteration. * The predicted value of the j-th leaf node.

[0069] c. Update the model

[0070] F m (x)=F m-, (x)+γ·h m (x)

[0071] F m (x) represents the model prediction value after the m-th iteration, F m-, (x) represents the model prediction value after the (m-1)th iteration, γ represents the learning rate, and h m (x) represents the predicted value of the decision tree model obtained from the m-th iteration of training.

[0072] d. Repeat steps (b)-(c) until the maximum number of iterations is reached, and sum the models to obtain the final prediction model.

[0073]

[0074] 2) Random Forest Model Training

[0075] First, a certain number of samples are randomly selected from the training dataset. For each sample, a certain number of features are randomly selected. Based on these features and samples, the CART (Classification and Regression Trees) algorithm is used to construct a decision tree. The CART algorithm recursively divides the dataset into smaller subsets to construct a binary tree-structured decision tree, where each internal node represents a decision condition for a feature, and each leaf node represents a numerical value. During the construction process, the CART algorithm minimizes the impurity of the partitioned dataset by selecting the optimal features and split points to achieve the best regression performance.

[0076] The CART algorithm uses mean squared error to measure the purity of the data and the importance of the features. The formula for calculating mean squared error is as follows:

[0077]

[0078] During training, the Random Forest algorithm continuously optimizes the splitting features and nodes of the decision trees to minimize the mean squared error, thereby constructing multiple efficient decision trees. After constructing multiple decision trees, the prediction results of each decision tree are averaged to obtain the final prediction result.

[0079] Specific steps:

[0080] a. Minimize the squared error to select the optimal feature and split point.

[0081]

[0082] R , R(j,s) and R1(j,s) represent subsets of features j that are greater than and less than the split point s, respectively. , c1 and c2 represent the average values ​​of the two subsets, respectively.

[0083] b. Calculation of predicted values

[0084] The predicted value is the average of the target variable for all samples at that node:

[0085]

[0086] c. Prediction results for each regression tree

[0087]

[0088] K is the number of leaf nodes in the m-th regression tree, γ m,* R is the predicted value of the i-th leaf node of the m-th regression tree. m,* It is the region of the i-th leaf node of the m-th regression tree, where IEx∈R m,* G is an indicator function.

[0089] d. Prediction results of the random forest model

[0090]

[0091] (5) Model prediction evaluation: The trained model is tested using a test set, and the mean squared error (MSE) and coefficient of determination (R²) of the model prediction results are calculated respectively. 2 The model performance is evaluated.

[0092]

[0093] In the formula, y * This is the actual value. This is an estimated value. This is the average of the actual values.

[0094] (6) Apply the trained prediction model to the detection of ammonia nitrogen concentration in actual water samples.

[0095] Application examples

[0096] Ammonia nitrogen and three-dimensional fluorescence data were analyzed from surface water and wastewater samples from river outlets. Ammonia nitrogen was detected using Nessler's reagent method. Three-dimensional fluorescence was performed using an excitation wavelength of 220–450 nm (Ex), a scan interval of 5 nm, and a slit width of 5 nm; an emission wavelength of 260–600 nm, a scan interval of 1 nm, and a slit width of 5 nm; and a scan rate of 2400 nm / min. A total of 116 water samples and blanks were obtained. 16027 fluorescence intensity characteristics were obtained for each water sample at different emission and excitation wavelengths. The size of the characteristic variable data was 116 × 16027. The ammonia nitrogen concentration distribution of the 116 water samples is shown in Table 1.

[0097] Table 1. Statistical analysis results of ammonia nitrogen concentration in water (unit: mg / L)

[0098]

[0099] After standardizing all the data, 80% was randomly selected as the training set, and the remaining 20% ​​was used as the test set for model training.

[0100] The gradient boosting tree and random forest algorithm toolkit from the scikit-learn library were used to train the model on the training set. In the gradient boosting regression tree model, the learning rate γ was set to 0.1, the number of trees was set to 100, the maximum depth was 3, the minimum number of samples required for node splits was 2, the minimum number of samples required for leaf nodes was 2, and the sampling ratio for each tree was 1.0 to prevent overfitting. Feature importance information was extracted during the model training process, ultimately yielding a machine learning algorithm model for predicting ammonia nitrogen concentration in water quality based on three-dimensional fluorescence intensity.

[0101] In the random forest model, there are 100 trees, the maximum depth of each tree is 3, the minimum number of samples required for node splitting is 2, the minimum number of samples required for leaf nodes is 1, and the feature sampling ratio of each tree is 1.0.

[0102] In the test set, the gradient boosting regression tree model showed R0 in its predictions. 2 The value is 0.952, and the MSE is 0.057. Random Forest model R... 2 The value is 0.800, and the MSE is 0.239. The gradient boosting model significantly outperforms the random forest model in predicting ammonia nitrogen concentration in water, and can accurately predict such concentrations.

[0103] Example 2

[0104] In the gradient boosting regression tree model, the learning rate γ is set to 0.1, the number of trees is set to 50, the maximum depth is 4, the minimum number of samples required for node splits is 2, the minimum number of samples required for leaf nodes is 2, the sampling ratio for each tree is 1.0, and other parameters are the same as in Example 1. The training model obtains prediction results on the test set, with R... 2 The value is 0.900, and the MSE is 0.120.

[0105] Example 3

[0106] In the gradient boosting regression tree model, the learning rate γ is set to 0.1, the number of trees is set to 100, the maximum depth is 2, the minimum number of samples required for node splits is 2, the minimum number of samples required for leaf nodes is 2, and the sampling ratio for each tree is 1.0. Other parameters are the same as in Example 1. The training model obtains prediction results on the test set, with R... 2 The value is 0.934, and the MSE is 0.079.

[0107] Comparative Example 1

[0108] In the random forest model, there are 100 trees, the maximum depth of each tree is 4, the minimum number of samples required for node splits is 2, the minimum number of samples required for leaf nodes is 1, and the feature sampling ratio of each tree is 1.0. The trained model R... 2 The value is 0.783, and the MSE is 0.260.

[0109] Comparative Example 2

[0110] In the random forest model, there are 100 trees, the maximum depth of each tree is 2, the minimum number of samples required for node splits is 2, the minimum number of samples required for leaf nodes is 1, and the feature sampling ratio of each tree is 1.0. The trained model R... 2 The value is 0.774, and the MSE is 0.271.

[0111] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A rapid detection method for ammonia nitrogen in water quality based on gradient boosting ensemble learning algorithm and three-dimensional fluorescence, characterized in that, include: (1) Collect three-dimensional fluorescence spectral data and ammonia nitrogen concentration data of water samples, preprocess them and construct a dataset, which includes a training dataset and a test dataset; (2) A gradient boosting regression tree model was constructed using the gradient boosting regression tree algorithm in the scikit-learn library, and trained using the training dataset to generate an ammonia nitrogen prediction model, including: (2-1) Define the training set , The number of training samples in the training set. For the three-dimensional fluorescence spectral data of the training samples, For the ammonia nitrogen concentration data of the training samples, The maximum number of iterations is given by the loss function. , To output a regression tree model; (2-2) Initialize the weak learner: in, It is a weak learner. It is a loss function; (2-3) Minimize the loss function: It is the first In the first iteration The predicted value of each leaf node. For the first The number of leaf nodes in a regression tree. It is a loss function. It is the first The true value of each sample It is the first The model after round of iterations for the 1st round The predicted value for each sample, No. The predicted values ​​for each leaf node; (2-4) Update the model: Indicates the first Model predictions after rounds of iterations Indicates the first Model predictions after rounds of iterations Indicates the learning rate. Indicates the first The predicted values ​​of the regression tree obtained from rounds of iterative training; (2-5) Repeat steps (2-3)-(2-4) until the maximum number of iterations is reached, and sum the models to obtain the final ammonia nitrogen prediction model: ； (3) Collect the three-dimensional fluorescence spectrum data of the water sample to be tested, preprocess it, and input it into the ammonia nitrogen prediction model to predict the ammonia nitrogen concentration of the water sample to be tested.

2. The rapid detection method for ammonia nitrogen in water quality based on gradient boosting ensemble learning algorithm and three-dimensional fluorescence as described in claim 1, characterized in that, When collecting three-dimensional fluorescence spectral matrix data of water samples, the detection conditions were as follows: excitation wavelength 220~450nm, scanning interval 5nm, slit width 5nm; emission wavelength 260~600nm, scanning interval 1nm, slit width 5nm; scanning speed 2400 nm / min.

3. The rapid detection method for ammonia nitrogen in water quality based on gradient boosting ensemble learning algorithm and three-dimensional fluorescence as described in claim 1, characterized in that, Z-score normalization was used to linearly transform the original matrix of the three-dimensional fluorescence spectral data and the ammonia nitrogen concentration data of the water sample to obtain normalized three-dimensional fluorescence spectral data and ammonia nitrogen concentration data.

4. The rapid detection method for ammonia nitrogen in water quality based on gradient boosting ensemble learning algorithm and three-dimensional fluorescence according to claim 3, characterized in that, The formula for linearly transforming the original matrix of the three-dimensional fluorescence spectral data and the ammonia nitrogen concentration data of the water sample using Z-score normalization is as follows: In the formula, These are the standardized data values. The original data values, The average of the original data values. This represents the standard deviation of the original data values.

5. The rapid detection method for ammonia nitrogen in water quality based on gradient boosting ensemble learning algorithm and three-dimensional fluorescence according to claim 1, characterized in that, When training a gradient boosting regression tree model, the learning rate Set the value to 0.1, the number of trees to 50-100, the maximum depth to 2-4, the minimum number of samples required for node splitting to be 2, the minimum number of samples required for leaf nodes to be 1, and the sampling ratio for each tree to be 1.

0.

6. The rapid detection method for ammonia nitrogen in water quality based on gradient boosting ensemble learning algorithm and three-dimensional fluorescence according to claim 5, characterized in that, When training a gradient boosting regression tree model, the learning rate The value is set to 0.1, the number of trees is set to 100, the maximum depth is 3, the minimum number of samples required for node splitting is 2, the minimum number of samples required for leaf nodes is 1, and the sampling ratio for each tree is 1.

0.

7. A rapid water quality ammonia nitrogen detection device based on gradient boosting ensemble learning algorithm and three-dimensional fluorescence, characterized in that, The aforementioned rapid ammonia nitrogen detection device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory stores an ammonia nitrogen prediction model established using the rapid ammonia nitrogen detection method according to any one of claims 1-6. When the processor executes the computer program, it performs the following steps: The three-dimensional fluorescence spectral data of the water sample to be tested is preprocessed and input into the ammonia nitrogen prediction model to predict the ammonia nitrogen concentration of the water sample and output the prediction results.

Images