Total nitrogen and total phosphorus concentration regional optimization prediction method and system

Abstract

The invention provides a total nitrogen and total phosphorus concentration regional optimization prediction method and system. The method comprises the following steps: collecting water quality on-line monitoring data, and carrying out data preprocessing on the water quality on-line monitoring data; area optimization processing: introducing random noise in a predefined range to the preprocessed water quality on-line monitoring data of the predefined area; performing clustering compression on the preprocessed water quality online monitoring data of the non-predefined area; constructing a total nitrogen and total phosphorus concentration monitoring model, and training the total nitrogen and total phosphorus concentration monitoring model by using the water quality online monitoring data subjected to regional optimization treatment; and performing total nitrogen and total phosphorus concentration prediction by using the trained total nitrogen and total phosphorus concentration monitoring model. According to the scheme provided by the invention, rapid total nitrogen and total phosphorus concentration online monitoring can be realized; the data weight of a research area is effectively expanded, noise interference is reduced, and the defect that a traditional method is insufficient in adaptability to a complex environment is overcome; and the monitoring precision of the total nitrogen and total phosphorus concentration is obviously improved.

Description

Technical Field

[0001] The present invention belongs to the field of total nitrogen and total phosphorus concentration monitoring in water bodies, and in particular relates to a method and system for regional optimization prediction of total nitrogen and total phosphorus concentrations. Background Art

[0002] Over the past few decades, rapid population growth, industrialization, and the expansion of urban areas have been causing water pollution worldwide. Large amounts of nutrients, such as nitrogen and phosphorus, have entered aquatic environments. This influx has accelerated the eutrophication process and led to a significant decline in water quality. Changes in nitrogen and phosphorus concentrations in rivers can have significant ecological impacts on reservoirs, nearby lakes, and coastal areas. Rapid, dynamic, and long-term monitoring of total nitrogen and total phosphorus concentrations in water bodies is crucial for monitoring and early warning of eutrophication and for the management and protection of aquatic ecosystems.

[0003] Automatic monitoring instruments for total nitrogen and total phosphorus mainly use ultraviolet absorption or chemiluminescence methods (Sun Zongguang et al., 2003). Measurements require collecting water samples, adding reagents, heating the reaction, and measuring light intensity. The measurement process is time-consuming, and the equipment is complex and expensive, which cannot meet the needs of rapid measurement.

[0004] There are some methods for retrieving total nitrogen and total phosphorus concentrations based on water reflectance spectra, also known as spectral methods (Liu Fuming, 2023). However, because total nitrogen and total phosphorus lack distinct spectral characteristics, their influence on the optical properties of water bodies is often indirect (Mathew et al., 2017; Dong et al., 2020). Relying solely on reflectance spectral remote sensing data to retrieve total nitrogen and total phosphorus concentrations has low accuracy.

[0005] Currently, the more mature online water quality monitoring equipment can quickly monitor water quality parameters including: water temperature, pH, dissolved oxygen, conductivity, turbidity, permanganate index, and ammonia nitrogen. However, the rapid monitoring of total nitrogen and total phosphorus concentrations has always been a problem.

[0006] Existing automatic monitoring instruments for total nitrogen and total phosphorus mainly use ultraviolet absorption or chemiluminescence methods, which require collecting water samples, adding reagents, heating the reaction, and measuring light intensity for measurement. The measurement process is time-consuming, and the equipment is complex and expensive, which cannot meet the needs of rapid measurement. Summary of the Invention

[0007] In order to solve the above technical problems, the present invention proposes a technical solution for a method for regional optimization prediction of total nitrogen and total phosphorus concentrations to solve the above technical problems.

[0008] The first aspect of the present invention discloses a method for regional optimization prediction of total nitrogen and total phosphorus concentrations, the method comprising:

[0009] Step S1: collecting water quality online monitoring data and performing data preprocessing on the water quality online monitoring data;

[0010] Step S2, regional optimization processing: introducing random noise within a predefined range into the preprocessed online water quality monitoring data of the predefined area; performing cluster compression on the preprocessed online water quality monitoring data of the non-predefined area;

[0011] Step S3: constructing a total nitrogen and total phosphorus concentration monitoring model, and using the water quality online monitoring data after regional optimization processing to train the total nitrogen and total phosphorus concentration monitoring model;

[0012] Step S4: collect water quality online monitoring data in real time, and input the data into the trained total nitrogen and total phosphorus concentration monitoring model after data preprocessing and regional optimization processing to predict the total nitrogen and total phosphorus concentrations.

[0013] According to the method of the first aspect of the present invention, in step S1, the water quality online monitoring data includes:

[0014] Water temperature, pH, dissolved oxygen, conductivity, turbidity, permanganate index, ammonia nitrogen, nitrate, chlorophyll, algae density and oil in water.

[0015] According to the method of the first aspect of the present invention, in step S1, the data preprocessing of the water quality online monitoring data includes:

[0016] Site condition outlier removal: retain data marked as "normal" for site conditions and delete data marked as "fault," "overrange," "debugging," and "verification and comparison."

[0017] Water quality monitoring outlier removal: Combined with time series analysis methods, the online water quality monitoring data after the site outliers are removed is used to detect fluctuations, identify rapidly changing values, and eliminate abnormal data that significantly deviates from the threshold range. At the same time, the triple standard deviation method is used to calculate the mean and standard deviation of the online water quality monitoring data. If a data point exceeds the range of 3 times the standard deviation of the mean, it is considered an outlier and removed.

[0018] Remove data points with incomplete water quality parameter data from online water quality monitoring data.

[0019] According to the method of the first aspect of the present invention, in step S2, introducing random noise within a predefined range into the preprocessed online water quality monitoring data of the predefined area includes:

[0020] Setting an error margin of ±0.1%-5%, randomly generating an error through uniform distribution, and then adding the error to the pre-processed water quality online monitoring data to generate monitoring data after the error is introduced;

[0021] The monitoring data after the error is introduced and the pre-processed online water quality monitoring data are repeatedly arranged and combined to generate monitoring data after regional data expansion.

[0022] According to the method of the first aspect of the present invention, in step S2, clustering and compressing the pre-processed online water quality monitoring data of the non-predefined area includes:

[0023] The K-means++ algorithm is used to initialize the cluster centers of the preprocessed online water quality monitoring data in non-predefined areas. The optimal number of clusters K is selected by combining the silhouette coefficient and the elbow rule to obtain the clustered monitoring data.

[0024] A stratified sampling strategy is used to compress the clustered monitoring data.

[0025] According to the method of the first aspect of the present invention, in step S3, the training of the total nitrogen and total phosphorus concentration monitoring model includes:

[0026] Hyperparameter optimization uses a grid search method to traverse and search different hyperparameter combinations to find the optimal configuration of the model:

[0027] Grid search finds the most suitable hyperparameter combination for the model by traversing a predefined hyperparameter grid; a preset range is set for each hyperparameter: the learning rate is set to 0.01 to 0.3; the maximum depth is set to an integer value from 3 to 10; and the number of decision trees is set to 50 to 1000.

[0028] According to the method of the first aspect of the present invention, in step S3, the training of the total nitrogen and total phosphorus concentration monitoring model further comprises:

[0029] A regularization method is used to impose penalties on the coefficients of the total nitrogen and total phosphorus concentration monitoring model;

[0030] The performance of the validation set in the online water quality monitoring data after optimization in the monitoring area is determined. If the performance of the validation set does not improve after multiple rounds of iterations, training is stopped early.

[0031] A second aspect of the present invention discloses a regional optimization prediction system for total nitrogen and total phosphorus concentrations, the system comprising:

[0032] The first processing module is configured to collect water quality online monitoring data and perform data preprocessing on the water quality online monitoring data;

[0033] The second processing module is configured to perform regional optimization processing: introducing random noise within a predefined range into the preprocessed online water quality monitoring data of the predefined area; and performing cluster compression on the preprocessed online water quality monitoring data of the non-predefined area;

[0034] The third processing module is configured to construct a total nitrogen and total phosphorus concentration monitoring model and train the total nitrogen and total phosphorus concentration monitoring model using the water quality online monitoring data after regional optimization processing;

[0035] The fourth processing module is configured to collect water quality online monitoring data in real time, and input the data into the trained total nitrogen and total phosphorus concentration monitoring model after data preprocessing and regional optimization processing to predict the total nitrogen and total phosphorus concentrations.

[0036] A third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it implements the steps of any one of the methods for regional optimization prediction of total nitrogen and total phosphorus concentrations described in the first aspect of the present disclosure.

[0037] A fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods for regional optimization prediction of total nitrogen and total phosphorus concentrations according to the first aspect of the present disclosure.

[0038] In summary, the solution proposed in the present invention can make full use of the water quality parameter data that can be quickly measured at present, and utilize the mutual influence mechanism of various water quality parameters in the water ecological environment system of the study area to complete the construction of the total nitrogen and total phosphorus concentration prediction model, which can realize rapid online monitoring of total nitrogen and total phosphorus concentrations; combine spatial and temporal factors with water quality parameters, add meteorological information, optimize the model input characteristics, establish a regional optimization processing method for water quality online monitoring data, effectively expand the data weight of the study area, reduce noise interference, and overcome the shortcomings of traditional methods in adapting to complex environments; based on the extreme gradient boosting method, fully explore the nonlinear relationship and feature interaction in water quality monitoring data, design a targeted hyperparameter optimization processing method, and significantly improve the accuracy of total nitrogen and total phosphorus concentration monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 This is a flow chart of a method for regional optimization prediction of total nitrogen and total phosphorus concentrations according to an embodiment of the present invention;

[0041] Figure 2This is a structural diagram of a regional optimization prediction system for total nitrogen and total phosphorus concentrations according to an embodiment of the present invention;

[0042] Figure 3 FIG. 4 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0044] The first aspect of the present invention discloses a method for regional optimization prediction of total nitrogen and total phosphorus concentrations. Figure 1 Flowchart of a method for regional optimization prediction of total nitrogen and total phosphorus concentrations according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0045] Step S1: collecting water quality online monitoring data and performing data preprocessing on the water quality online monitoring data;

[0046] Step S2, regional optimization processing: introducing random noise within a predefined range into the preprocessed online water quality monitoring data of the predefined area; performing cluster compression on the preprocessed online water quality monitoring data of the non-predefined area;

[0047] Step S3: constructing a total nitrogen and total phosphorus concentration monitoring model, and using the water quality online monitoring data after regional optimization processing to train the total nitrogen and total phosphorus concentration monitoring model;

[0048] Step S4: collect water quality online monitoring data in real time, and input the data into the trained total nitrogen and total phosphorus concentration monitoring model after data preprocessing and regional optimization processing to predict the total nitrogen and total phosphorus concentrations.

[0049] In step S1, water quality online monitoring data is collected and preprocessed.

[0050] In some embodiments, in step S1, the water quality online monitoring data includes:

[0051] Water temperature, pH, dissolved oxygen, conductivity, turbidity, permanganate index, ammonia nitrogen, nitrate, chlorophyll, algae density and oil in water.

[0052] The data preprocessing of the water quality online monitoring data includes:

[0053] Removal of outliers at site conditions: Water quality online monitoring stations may experience some data deviations from the normal range due to equipment failure, maintenance, or environmental interference. To ensure data accuracy and reliability, data marked as "normal" at the site condition is retained, and data marked as "fault," "overrange," "debugging," and "verification and comparison" are deleted.

[0054] Removal of outliers in water quality monitoring: Combined with time series analysis methods (such as sliding window method, Z-score detection, etc.), the online water quality monitoring data after the site outliers are removed is subjected to volatility detection, rapidly changing values ​​are identified, and abnormal data that significantly deviate from the threshold range are eliminated; at the same time, the triple standard deviation method is used to calculate the mean and standard deviation of the online water quality monitoring data. If a data point exceeds the range of 3 times the standard deviation of the mean, the data point is considered to be an outlier and is removed; specifically, the water quality parameters are tested one by one, and outliers that are too high or too low and do not conform to reality are eliminated to reduce the interference of noise on data analysis.

[0055] Removing data points with incomplete water quality parameter data in online water quality monitoring data: Since some monitoring points have missing data, the present invention only retains records with valid data for all water quality parameters and eliminates those points with monitoring data for only some indicators to ensure the integrity of subsequent analysis.

[0056] In step S2, regional optimization processing is performed: random noise within a predefined range is introduced into the preprocessed online water quality monitoring data of the predefined area; and cluster compression is performed on the preprocessed online water quality monitoring data of the non-predefined area.

[0057] In some embodiments, in step S2, to address the problems of data redundancy and low training efficiency caused by excessive global data volume in the training of the online water quality monitoring model, as well as limited model accuracy caused by insufficient data in the predefined area, random noise within a predefined range is introduced into the preprocessed online water quality monitoring data of the predefined area to expand the effective sample size of the data set, improve the generalization ability of the model, and avoid overfitting, including:

[0058] Setting an error boundary of ±1-105%, randomly generating an error through uniform distribution, and then adding the error to the pre-processed water quality online monitoring data to generate monitoring data after the error is introduced;

[0059] The monitoring data after the error is introduced and the pre-processed online water quality monitoring data are repeatedly arranged and combined to generate monitoring data after regional data expansion.

[0060] The above method can not only expand the sample size of the data set, but also enhance the robustness of the model, reduce the overfitting problem caused by insufficient data, and reduce the impact of noise on the prediction results.

[0061] To address the problem of low training efficiency caused by excessive global data volume, clustering and compression of pre-processed online water quality monitoring data in non-predefined areas are performed, including:

[0062] The K-means++ algorithm is used to initialize the cluster centers of the preprocessed online water quality monitoring data in non-predefined areas. The optimal number of clusters K is selected by combining the silhouette coefficient and the elbow rule to obtain the clustered monitoring data.

[0063] A stratified sampling strategy is used to compress the clustered monitoring data. Specifically, samples are randomly drawn proportionally from each cluster subset. The sampling ratio is set based on the actual data distribution to ensure that the differences and important characteristics between clusters are preserved while compressing the data volume. This reduces redundant data, improves training efficiency, and avoids information loss caused by excessive data compression.

[0064] In step S3, a total nitrogen and total phosphorus concentration monitoring model is constructed, and the water quality online monitoring data after regional optimization processing is used to train the total nitrogen and total phosphorus concentration monitoring model.

[0065] In some embodiments, in step S3, training the total nitrogen and total phosphorus concentration monitoring model includes:

[0066] Hyperparameter optimization uses a grid search method to traverse and search different hyperparameter combinations to find the optimal configuration of the model:

[0067] Grid search finds the most suitable hyperparameter combination for the model by traversing a predefined hyperparameter grid; a preset range is set for each hyperparameter: the learning rate is set to 0.01 to 0.3; the maximum depth is set to an integer value from 3 to 10; and the number of decision trees is set to 50 to 1000.

[0068] The training of the total nitrogen and total phosphorus concentration monitoring model further comprises:

[0069] A regularization method is used to impose penalties on the coefficients of the total nitrogen and total phosphorus concentration monitoring model;

[0070] The performance of the validation set in the online water quality monitoring data after optimization in the monitoring area is determined. If the performance of the validation set does not improve after multiple rounds of iterations, training is stopped early.

[0071] Specifically, the dataset production takes into account the temporal and spatial distribution attribute characteristic parameters and meteorological information of the study area, including the location of the observation site, water body type, measurement date, and weather conditions, so as to enhance the total nitrogen and total phosphorus concentration monitoring model's ability to identify regional characteristics and seasonal changes.

[0072] Based on the principle of the extreme gradient boosting tree of artificial intelligence machine learning model and the training data set prepared above, the present invention constructs a total nitrogen and total phosphorus concentration monitoring model. The specific model construction process is as follows:

[0073] (1) Data division and preprocessing: The data set is divided into a training set and a test set in a ratio of 8:2, of which 80% of the data is used for model training and verification, and 20% of the data is used for model accuracy testing. The input feature parameters (such as water quality indicators such as water temperature and pH, and spatiotemporal and meteorological information of the study area) are standardized. In order to improve the training effect of the model, the input features are Z-score standardized, that is:

[0074]

[0075] Here, μ is the mean and σ is the standard deviation, which ensures that the mean of all input features is 0 and the standard deviation is 1, eliminating the impact of dimensional differences on model training.

[0076] In addition, to reduce the influence of the skewed distribution of the target variables, the model output (total nitrogen and total phosphorus concentrations) was subjected to logarithmic (log10) spatial transformation, i.e.

[0077] Y′=log 10 (Y) (2)

[0078] Where Y is the total nitrogen or total phosphorus concentration, and Y′ is the corresponding value after log10 transformation. This transformation can reduce the long-tail effect of the data and thus improve the model fitting effect.

[0079] (2) Hyperparameter tuning: Use the training set to train and validate the model, and use the 5-fold cross-validation method to tune the model's hyperparameters. The cross-validation steps are as follows:

[0080] The training set is randomly divided into five equal or nearly equal-sized subsets (folds). In each iteration, four of these folds are selected as training sets for model training, while the remaining fold is used as a validation set for model performance evaluation. This process is repeated five times to ensure that each fold can be tested as a validation set, and the validation results of all folds are recorded. Finally, the average of the five validation results is calculated as a comprehensive evaluation indicator of model performance.

[0081] Hyperparameter optimization uses the Grid Search method to traverse and search different hyperparameter combinations to find the optimal configuration of the model. Grid Search finds the most suitable hyperparameter combination for the model by traversing the predefined hyperparameter grid. Taking XGBoost as an example, a reasonable range is set for each hyperparameter, such as: the learning rate is set from 0.01 to 0.53, and different step lengths are tested; the maximum depth is set to an integer value from 3 to 10, covering common tree depth choices; the number of decision trees is set from 50 to 1000, depending on the model complexity and data volume. For each hyperparameter combination, 5-fold cross-validation is used, and the corresponding evaluation indicators are calculated;

[0082] During the model tuning process, overfitting is prone to occur, especially when the model's hyperparameters, such as the depth of the tree, are too large, or when some hyperparameters are set improperly during the model training process. To this end, the present invention adopts the following strategies: Regularization. In the XGBoost model, the L1 (Lasso) and L2 (Ridge) regularization methods can effectively constrain the model complexity and avoid overfitting of the model to the training data by imposing penalties on the model's coefficients. Early Stopping: monitor the performance of the validation set during training. If the performance of the validation set no longer improves after multiple rounds of iterations, stop training in advance to avoid overfitting.

[0083] Finally, by comparing the verification results of all hyperparameter combinations, the hyperparameter combination with the best performance is selected as the final model configuration, and it is ensured that the model has good generalization ability under different data segmentation conditions.

[0084] Final model training and testing: After determining the model structure and hyperparameters, final model training is performed using the complete training set to fully exploit the information in the training data and obtain the best-performing estimation model. The model's accuracy is evaluated on the test set, using multiple performance metrics to verify the model's estimation ability and stability on unknown data.

[0085] This embodiment selects multiple quantitative evaluation indicators to comprehensively evaluate the accuracy of the machine learning model, including root mean square error RMSE (mg / L), mean absolute error MAE (mg / L), mean relative error MAPE, and mean error Bias (mg / L).

[0086]

[0087] Where, E i is the estimated value of the model, M i is the true value of the measurement, and n is the number of samples. MAE and Bias are both values ​​in the logarithmic transformation space, Z is the middle value, and Y is the middle value for the convenience of calculation.

[0088] In step S4, water quality online monitoring data is collected in real time, and after data preprocessing and regional optimization processing, it is input into the trained total nitrogen and total phosphorus concentration monitoring model to predict the total nitrogen and total phosphorus concentrations.

[0089] Specifically, the input data required for the application of the total nitrogen and total phosphorus concentration monitoring model are the online water quality monitoring data, as well as the spatiotemporal distribution and meteorological information of the study area. The final output of the model is the estimated total nitrogen and total phosphorus concentrations in the water body.

[0090] In summary, the solution proposed in the present invention can make full use of the water quality parameter data that can be quickly measured at present, and utilize the mutual influence mechanism of various water quality parameters in the water ecological environment system of the study area to complete the construction of the total nitrogen and total phosphorus concentration prediction model, which can realize rapid online monitoring of total nitrogen and total phosphorus concentrations; combine spatial and temporal factors with water quality parameters, add meteorological information, optimize the model input characteristics, establish a regional optimization processing method for water quality online monitoring data, effectively expand the data weight of the study area, reduce noise interference, and overcome the shortcomings of traditional methods in adapting to complex environments; based on the extreme gradient boosting method, fully explore the nonlinear relationship and feature interaction in water quality monitoring data, design a targeted hyperparameter optimization processing method, and significantly improve the accuracy of total nitrogen and total phosphorus concentration monitoring.

[0091] The second aspect of the present invention discloses a regional optimization prediction system for total nitrogen and total phosphorus concentrations. Figure 2 FIG. 1 is a structural diagram of a regional optimization prediction system for total nitrogen and total phosphorus concentrations according to an embodiment of the present invention; Figure 2 As shown, the system 100 includes:

[0092] The first processing module 101 is configured to collect water quality online monitoring data and perform data preprocessing on the water quality online monitoring data;

[0093] The second processing module 102 is configured to perform regional optimization processing: introducing random noise within a predefined range to the preprocessed online water quality monitoring data of the predefined area; and performing cluster compression on the preprocessed online water quality monitoring data of the non-predefined area;

[0094] The third processing module 103 is configured to construct a total nitrogen and total phosphorus concentration monitoring model and train the total nitrogen and total phosphorus concentration monitoring model using the water quality online monitoring data after regional optimization processing;

[0095] The fourth processing module 104 is configured to collect water quality online monitoring data in real time, and input the trained total nitrogen and total phosphorus concentration monitoring model after data preprocessing and regional optimization processing to predict total nitrogen and total phosphorus concentrations.

[0096] According to the system of the second aspect of the present invention, the first processing module 101 is specifically configured such that the water quality online monitoring data includes:

[0097] Water temperature, pH, dissolved oxygen, conductivity, turbidity, permanganate index, ammonia nitrogen, nitrate, chlorophyll, algae density and oil in water.

[0098] The data preprocessing of the water quality online monitoring data includes:

[0099] Removal of outliers at site conditions: Water quality online monitoring stations may experience some data deviations from the normal range due to equipment failure, maintenance, or environmental interference. To ensure data accuracy and reliability, data marked as "normal" at the site condition is retained, and data marked as "fault," "overrange," "debugging," and "verification and comparison" are deleted.

[0100] Removal of outliers in water quality monitoring: Combined with time series analysis methods (such as sliding window method, Z-score detection, etc.), the online water quality monitoring data after the site outliers are removed is subjected to volatility detection, rapidly changing values ​​are identified, and abnormal data that significantly deviate from the threshold range are eliminated; at the same time, the triple standard deviation method is used to calculate the mean and standard deviation of the online water quality monitoring data. If a data point exceeds the range of 3 times the standard deviation of the mean, the data point is considered to be an outlier and is removed; specifically, the water quality parameters are tested one by one, and outliers that are too high or too low and do not conform to reality are eliminated to reduce the interference of noise on data analysis.

[0101] Removing data points with incomplete water quality parameter data in online water quality monitoring data: Since some monitoring points have missing data, the present invention only retains records with valid data for all water quality parameters and eliminates those points with monitoring data for only some indicators to ensure the integrity of subsequent analysis.

[0102] According to the system of the second aspect of the present invention, the second processing module 102 is specifically configured to, in order to address the problems of data redundancy and low training efficiency caused by an excessive amount of global data in the training of the online water quality monitoring model, and limited model accuracy caused by insufficient data in the predefined area, introduce random noise within a predefined range into the preprocessed online water quality monitoring data of the predefined area, thereby expanding the effective sample size of the data set, improving the generalization ability of the model, and avoiding overfitting, including:

[0103] Setting an error boundary of ±1-105%, randomly generating an error through uniform distribution, and then adding the error to the pre-processed water quality online monitoring data to generate monitoring data after the error is introduced;

[0104] The monitoring data after the error is introduced and the pre-processed online water quality monitoring data are repeatedly arranged and combined to generate monitoring data after regional data expansion.

[0105] The above method can not only expand the sample size of the data set, but also enhance the robustness of the model, reduce the overfitting problem caused by insufficient data, and reduce the impact of noise on the prediction results.

[0106] To address the problem of low training efficiency caused by excessive global data volume, clustering and compression of pre-processed online water quality monitoring data in non-predefined areas are performed, including:

[0107] The K-means++ algorithm is used to initialize the cluster centers of the preprocessed online water quality monitoring data in non-predefined areas. The optimal number of clusters K is selected by combining the silhouette coefficient and the elbow rule to obtain the clustered monitoring data.

[0108] A stratified sampling strategy is used to compress the clustered monitoring data. Specifically, samples are randomly drawn proportionally from each cluster subset. The sampling ratio is set based on the actual data distribution to ensure that the differences and important characteristics between clusters are preserved while compressing the data volume. This reduces redundant data, improves training efficiency, and avoids information loss caused by excessive data compression.

[0109] According to the system of the second aspect of the present invention, the third processing module 103 is specifically configured as follows: the training of the total nitrogen and total phosphorus concentration monitoring model includes:

[0110] Hyperparameter optimization uses a grid search method to traverse and search different hyperparameter combinations to find the optimal configuration of the model:

[0111] Grid search finds the most suitable hyperparameter combination for the model by traversing a predefined hyperparameter grid; a preset range is set for each hyperparameter: the learning rate is set to 0.01 to 0.3; the maximum depth is set to an integer value from 3 to 10; and the number of decision trees is set to 50 to 1000.

[0112] The training of the total nitrogen and total phosphorus concentration monitoring model further comprises:

[0113] A regularization method is used to impose penalties on the coefficients of the total nitrogen and total phosphorus concentration monitoring model;

[0114] The performance of the validation set in the online water quality monitoring data after optimization in the monitoring area is determined. If the performance of the validation set does not improve after multiple rounds of iterations, training is stopped early.

[0115] Specifically, the dataset production takes into account the temporal and spatial distribution attribute characteristic parameters and meteorological information of the study area, including the location of the observation site, water body type, measurement date, and weather conditions, so as to enhance the total nitrogen and total phosphorus concentration monitoring model's ability to identify regional characteristics and seasonal changes.

[0116] Based on the principle of the extreme gradient boosting tree of artificial intelligence machine learning model and the training data set prepared above, the present invention constructs a total nitrogen and total phosphorus concentration monitoring model. The specific model construction process is as follows:

[0117] (1) Data partitioning and preprocessing: The dataset was divided into a training set and a test set in a ratio of 8:2, with 80% of the data used for model training and validation, and 20% for model accuracy testing. Input feature parameters (such as water quality indicators such as water temperature and pH, and temporal, spatial, and meteorological information of the study area) were standardized. To improve the training effect of the model, the input features were normalized using the Z-score.

[0118] In addition, to reduce the impact of the skewed distribution of the target variables, the model output (total nitrogen and total phosphorus concentrations) was transformed into a logarithmic (log10) space. This transformation can reduce the long-tail effect of the data and thus improve the model fitting effect.

[0119] (2) Hyperparameter tuning: Use the training set to train and validate the model, and use the 5-fold cross-validation method to tune the model's hyperparameters. The cross-validation steps are as follows:

[0120] The training set is randomly divided into five equal or nearly equal-sized subsets (folds). In each iteration, four of these folds are selected as training sets for model training, while the remaining fold is used as a validation set for model performance evaluation. This process is repeated five times to ensure that each fold can be tested as a validation set, and the validation results of all folds are recorded. Finally, the average of the five validation results is calculated as a comprehensive evaluation indicator of model performance.

[0121] Hyperparameter optimization uses the Grid Search method to traverse and search different hyperparameter combinations to find the optimal configuration of the model. Grid Search finds the most suitable hyperparameter combination for the model by traversing the predefined hyperparameter grid. Taking XGBoost as an example, a reasonable range is set for each hyperparameter, such as: the learning rate is set from 0.01 to 0.3, and different step lengths are tested; the maximum depth is set to an integer value from 3 to 10, covering common tree depth choices; the number of decision trees is set from 50 to 1000, depending on the model complexity and data volume. For each hyperparameter combination, 5-fold cross-validation is used, and the corresponding evaluation indicators are calculated;

[0122] During the model tuning process, overfitting is prone to occur, especially when the model's hyperparameters, such as the depth of the tree, are too large, or when some hyperparameters are set improperly during the model training process. To this end, the present invention adopts the following strategies: Regularization. In the XGBoost model, the L1 (Lasso) and L2 (Ridge) regularization methods can effectively constrain the model complexity and avoid overfitting of the model to the training data by imposing penalties on the model's coefficients. Early Stopping: monitor the performance of the validation set during training. If the performance of the validation set no longer improves after multiple rounds of iterations, stop training in advance to avoid overfitting.

[0123] Finally, by comparing the verification results of all hyperparameter combinations, the hyperparameter combination with the best performance is selected as the final model configuration, and it is ensured that the model has good generalization ability under different data segmentation conditions.

[0124] Final model training and testing: After determining the model structure and hyperparameters, final model training is performed using the complete training set to fully exploit the information in the training data and obtain the best-performing estimation model. The model's accuracy is evaluated on the test set, using multiple performance metrics to verify the model's estimation ability and stability on unknown data.

[0125] This embodiment selects multiple quantitative evaluation indicators to comprehensively evaluate the accuracy of the machine learning model, including root mean square error RMSE (mg / L), mean absolute error MAE (mg / L), mean relative error MAPE, and mean error Bias (mg / L).

[0126] According to the system of the second aspect of the present invention, the fourth processing module 104 is specifically configured such that the input data required for the application of the total nitrogen and total phosphorus concentration monitoring model is the online water quality monitoring data, as well as the spatiotemporal distribution and meteorological information of the study area, and the final output result of the model is the estimated total nitrogen and total phosphorus concentrations in the water body.

[0127] A third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of any one of the methods for regional optimization prediction of total nitrogen and total phosphorus concentrations disclosed in the first aspect of the present invention.

[0128] Figure 3 FIG. 1 is a structural diagram of an electronic device according to an embodiment of the present invention. Figure 3As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the electronic device housing, or an external keyboard, touchpad or mouse.

[0129] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a structural diagram of the part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0130] A fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the steps of any one of the methods for regional optimization prediction of total nitrogen and total phosphorus concentrations disclosed in the first aspect of the present invention.

[0131] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above embodiments only express several implementation methods of the present application. The description is relatively specific and detailed, but it cannot be understood as a limitation on the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of this application, several variations and improvements can be made, which all fall within the scope of protection of this application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. A method for regional optimization prediction of total nitrogen and total phosphorus concentrations, characterized in that: The method comprises: Step S1: collecting water quality online monitoring data and performing data preprocessing on the water quality online monitoring data; Step S2, regional optimization processing: introducing random noise within a predefined range into the preprocessed online water quality monitoring data of the predefined area; performing cluster compression on the preprocessed online water quality monitoring data of the non-predefined area; Step S3: constructing a total nitrogen and total phosphorus concentration monitoring model, and using the water quality online monitoring data after regional optimization processing to train the total nitrogen and total phosphorus concentration monitoring model; Step S4: collect water quality online monitoring data in real time, and input the data into the trained total nitrogen and total phosphorus concentration monitoring model after data preprocessing and regional optimization processing to predict the total nitrogen and total phosphorus concentrations.

2. A method for regional optimization prediction of total nitrogen and total phosphorus concentrations according to claim 1, characterized in that: In step S1, the water quality online monitoring data includes: Water temperature, pH, dissolved oxygen, conductivity, turbidity, permanganate index, ammonia nitrogen, nitrate, chlorophyll, algae density and oil in water.

3. A method for regional optimization prediction of total nitrogen and total phosphorus concentrations according to claim 1, characterized in that: In step S1, the data preprocessing of the water quality online monitoring data includes: Removal of site condition outliers: retain data marked as "normal" for site conditions and delete data marked as "fault," "overrange," "debugging," and "verification and comparison." Water quality monitoring outlier removal: Combined with time series analysis methods, the online water quality monitoring data after the site outliers are removed is used to detect fluctuations, identify rapidly changing values, and eliminate abnormal data that significantly deviates from the threshold range. At the same time, the triple standard deviation method is used to calculate the mean and standard deviation of the online water quality monitoring data. If a data point exceeds the range of 3 times the standard deviation of the mean, it is considered an outlier and removed. Remove data points with incomplete water quality parameter data from online water quality monitoring data.

4. A method for regional optimization prediction of total nitrogen and total phosphorus concentrations according to claim 1, characterized in that: In step S2, the introduction of random noise within a predefined range into the preprocessed online water quality monitoring data of the predefined area includes: Setting an error margin of ±0.1% to 5%, randomly generating an error through uniform distribution, and then adding the error to the pre-processed online water quality monitoring data to generate monitoring data after the error is introduced; The monitoring data after the error is introduced and the pre-processed online water quality monitoring data are repeatedly arranged and combined to generate monitoring data after regional data expansion.

5. A method for regional optimization prediction of total nitrogen and total phosphorus concentrations according to claim 1, characterized in that: In step S2, clustering and compressing the pre-processed online water quality monitoring data of the non-predefined area includes: The K-means++ algorithm is used to initialize the cluster centers of the preprocessed online water quality monitoring data in non-predefined areas. The optimal number of clusters K is selected by combining the silhouette coefficient and the elbow rule to obtain the clustered monitoring data. A stratified sampling strategy is used to compress the clustered monitoring data.

6. A method for regional optimization prediction of total nitrogen and total phosphorus concentrations according to claim 1, characterized in that: In step S3, the training of the total nitrogen and total phosphorus concentration monitoring model includes: Hyperparameter optimization uses a grid search method to traverse and search different hyperparameter combinations to find the optimal configuration of the model: Grid search finds the most suitable hyperparameter combination for the model by traversing a predefined hyperparameter grid; a preset range is set for each hyperparameter: the learning rate is set to 0.01 to 0.3; the maximum depth is set to an integer value from 3 to 10; and the number of decision trees is set to 50 to 1000.

7. The method for regional optimization prediction of total nitrogen and total phosphorus concentrations according to claim 1, characterized in that: In step S3, the training of the total nitrogen and total phosphorus concentration monitoring model further includes: A regularization method is used to impose penalties on the coefficients of the total nitrogen and total phosphorus concentration monitoring model; The performance of the validation set in the online water quality monitoring data after optimization in the monitoring area is determined. If the performance of the validation set does not improve after multiple rounds of iterations, training is stopped early.

8. A regional optimization prediction system for total nitrogen and total phosphorus concentrations, characterized in that: The system comprises: The first processing module is configured to collect water quality online monitoring data and perform data preprocessing on the water quality online monitoring data; The second processing module is configured to perform regional optimization processing: introducing random noise within a predefined range into the preprocessed online water quality monitoring data of the predefined area; and performing cluster compression on the preprocessed online water quality monitoring data of the non-predefined area; The third processing module is configured to construct a total nitrogen and total phosphorus concentration monitoring model and train the total nitrogen and total phosphorus concentration monitoring model using the water quality online monitoring data after regional optimization processing; The fourth processing module is configured to collect water quality online monitoring data in real time, and input the data into the trained total nitrogen and total phosphorus concentration monitoring model after data preprocessing and regional optimization processing to predict the total nitrogen and total phosphorus concentrations.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the steps in the method for regional optimization prediction of total nitrogen and total phosphorus concentrations according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method for regional optimization prediction of total nitrogen and total phosphorus concentrations according to any one of claims 1 to 7 are implemented.

Images