Surface water monitoring data completion and credibility evaluation method based on machine learning

Abstract

This application discloses a machine learning-based method for surface water monitoring data completion and reliability assessment, relating to the interdisciplinary fields of water environment monitoring and artificial intelligence. The method includes the following steps: processing historical multi-parameter time-series data from the acquired surface water monitoring network to obtain filtered features; obtaining a completion model and supporting rules based on the filtered features; acquiring real-time multi-parameter time-series data from the surface water monitoring network; applying the supporting rules to process the real-time multi-parameter time-series data to form a real-time sample set to be completed; matching the current completion model and outputting completed data; calculating the completion error between the completed data and the true values; determining the reliability level using a tiered threshold method and outputting an assessment report; and incrementally training the completion model according to preset update conditions to obtain an updated completion model. This application can achieve high-precision completion and reliability assessment of surface water monitoring data, effectively improving the data quality of the surface water monitoring network.

Description

Technical Field

[0001] This application relates to the interdisciplinary field of water environment monitoring and artificial intelligence, and in particular to a method for completing and assessing the credibility of surface water monitoring data based on machine learning. Background Technology

[0002] Surface water monitoring networks collect water quality parameters such as pH, dissolved oxygen, and chemical oxygen demand (COD) through the deployment of numerous sensors, providing crucial data support for water environment management. However, due to factors such as sensor failure, communication interruptions, and extreme weather, monitoring data is prone to loss, with a loss rate reaching 15%-30%, severely reducing the reliability of subsequent data analysis and decision-making.

[0003] Existing methods for handling missing data have significant limitations: (1) Traditional statistical methods (such as mean filling and linear interpolation) do not consider the spatiotemporal correlation of water quality parameters, resulting in low accuracy of completion, and are particularly ineffective for continuous missing data exceeding 24 hours.

[0004] (2) Single machine learning models (such as BP neural network and support vector machine) are difficult to capture the spatial coupling relationship and time series dynamic features between multiple monitoring points, and the completion error exceeds 10% in complex watershed scenarios.

[0005] (3) Existing technologies lack a credibility assessment mechanism for the completion results, which cannot quantify the reliability of the completion data. This leads to decision-making risks in subsequent applications such as water quality assessment and pollution early warning. Furthermore, no differentiated processing strategies have been developed for different missing modes such as random missing, continuous missing, and systematic missing, resulting in poor universality.

[0006] Therefore, there is an urgent need to develop a surface water monitoring data processing method that adapts to different missing patterns, integrates multi-dimensional features, and has both completion accuracy and reliability assessment capabilities. Summary of the Invention

[0007] The purpose of this application is to provide a machine learning-based method for surface water monitoring data completion and reliability assessment, which can achieve high-precision completion and reliability assessment of surface water monitoring data, and effectively improve the data quality of surface water monitoring networks.

[0008] To achieve the above objectives, this application provides a machine learning-based method for surface water monitoring data completion and reliability assessment, comprising the following steps: S1: Acquire historical multi-parameter time-series data of the surface water monitoring network, and sequentially perform preprocessing, missing pattern identification, multi-source spatiotemporal feature extraction and filtering on the historical multi-parameter time-series data to obtain filtered features; configure corresponding ensemble learning frameworks for three types of missing patterns: random missing, continuous missing, and systematic missing; train and validate the ensemble learning frameworks based on the filtered features to obtain a completion model and supporting rules, wherein the supporting rules include: preprocessing rules, feature extraction rules, and missing pattern identification rules; S2: Acquire real-time multi-parameter time-series data of the surface water monitoring network, and apply the supporting rules to the real-time multi-parameter time-series data for further processing. S3: Perform preprocessing, multi-source spatiotemporal feature extraction, and missing pattern recognition, and label the current missing pattern of each missing sample to form a real-time imputation sample set; S4: Match the corresponding imputation model as the current imputation model according to the current missing pattern of each missing sample in the real-time imputation sample set, input the real-time imputation sample set into the current imputation model, and output the imputation data from the current imputation model; S5: Obtain the true value used to verify the imputation data, calculate the imputation error between the imputation data and the true value, analyze the imputation error using the hierarchical threshold method, determine the credibility level of the imputation data, and output an evaluation report containing the imputation data, imputation error, and credibility level; S6: Perform incremental training on the imputation model using newly added historical complete data according to the preset update conditions to obtain the updated imputation model.

[0009] As described above, the sub-steps for obtaining historical multi-parameter time-series data from the surface water monitoring network, performing preprocessing, missing pattern identification, multi-source spatiotemporal feature extraction and filtering on the historical multi-parameter time-series data, and obtaining the filtered features are as follows: S11: Obtain historical multi-parameter time-series data from the surface water monitoring network, preprocess the historical multi-parameter time-series data, and convert the historical multi-parameter time-series data into dimensionless data in the [0,1] interval; wherein, the historical multi-parameter time-series data includes at least the time series of 8 core water quality parameters: pH value, dissolved oxygen, turbidity, chemical oxygen demand, ammonia nitrogen, total phosphorus, total nitrogen and permanganate index. Data; S12: Identify historical missing patterns based on the duration, distribution characteristics, and data correlation of dimensionless data; where historical missing patterns include any one or more of random missing, continuous missing, and systematic missing; S13: Extract multi-source spatiotemporal features from dimensionless data to obtain historical multi-source spatiotemporal features; where historical multi-source spatiotemporal features include time dimension features and spatial dimension features; S14: Use a recursive feature elimination algorithm, combined with feature importance ranking, to filter historical multi-source spatiotemporal features, obtaining historical multi-source spatiotemporal features with a cumulative contribution exceeding the contribution threshold as the filtered features.

[0010] As described above, the sub-steps for using a recursive feature elimination algorithm combined with feature importance ranking to filter historical multi-source spatiotemporal features and obtain historical multi-source spatiotemporal features with a cumulative contribution exceeding a contribution threshold as the filtered features are as follows: S141: Construct an initial contribution analysis model based on historical multi-source spatiotemporal features. Train the initial contribution analysis model using complete water quality data selected from historical multi-parameter time series data to obtain the contribution analysis model. Quantify the contribution of each historical multi-source spatiotemporal feature to the water quality parameter prediction results through the feature importance evaluation mechanism built into the contribution analysis model. Generate a feature importance ranking table from high to low contribution, completing the construction of the feature importance ranking. S142: Use the features in the feature importance ranking table as the set to be filtered and set the contribution threshold. Use the set to be filtered and the contribution threshold as the initialized filtering parameters. S143: Select the first historical multi-source spatiotemporal feature not in the added feature list from the set to be filtered, according to the ascending order of the feature importance ranking table. Add this historical multi-source spatiotemporal feature to the temporary feature subset and simultaneously add it to the added feature list. Then, use the updated temporary feature subset as the current temporary feature subset to input the contribution analysis model for training and calculate the cumulative contribution of the current temporary feature subset. S144: Determine whether the cumulative contribution has reached the contribution threshold. If not, continue to select the next historical multi-source spatiotemporal feature not in the added feature list from the set to be filtered, according to the ascending order of the feature importance ranking table, and repeat S143. If the threshold is reached, immediately stop adding historical multi-source spatiotemporal features, use the current temporary feature subset as the filtered features, and discard the remaining unadded historical multi-source spatiotemporal features in the set to be filtered.

[0011] As described above, the sub-steps for configuring corresponding ensemble learning frameworks for the three missing data patterns—random missing, continuous missing, and systematic missing—and training and validating the ensemble learning frameworks based on the selected features to obtain the completion models and supporting rules are as follows: S11': Configure corresponding ensemble learning frameworks for the three missing data patterns—random missing, continuous missing, and systematic missing—for each; S12': Train and validate the ensemble learning frameworks for the three missing data patterns based on the selected features to obtain the completion models for the three missing data patterns; S13': Use the 3σ principle or IQR method to handle outliers, standardize multi-dimensional features, and unify time series timestamps to achieve time alignment as preprocessing rules; use spatial features extracted based on monitoring station locations, temporal features extracted based on collection time, calculate statistical features, and construct feature interaction terms as feature extraction rules; use missing value distribution analysis and statistical tests to identify random missing data, use time series analysis to identify continuous missing data, and use pattern matching and cluster analysis to identify systematic missing data as missing data pattern recognition rules; and use the preprocessing rules, feature extraction rules, and missing data pattern recognition rules together as supporting rules.

[0012] As described above, the sub-steps for acquiring real-time multi-parameter time-series data from the surface water monitoring network, applying supporting rules to preprocess the real-time multi-parameter time-series data, extract multi-source spatiotemporal features and identify missing patterns, and labeling the current missing pattern of each missing sample to form a real-time sample set to be completed are as follows: S21: Acquire real-time multi-parameter time-series data from the surface water monitoring network, wherein the real-time multi-parameter time-series data includes at least the time-series data of the following eight core water quality parameters: pH, dissolved oxygen, turbidity, chemical oxygen demand, ammonia nitrogen, total phosphorus, total nitrogen and permanganate index; S S22: Apply the preprocessing rules in the supporting rules to preprocess the real-time multi-parameter time series data to obtain real-time dimensionless data; S23: Apply the feature extraction rules in the supporting rules to extract multi-source spatiotemporal features from the real-time dimensionless data to obtain real-time multi-source spatiotemporal features; among which, the real-time multi-source spatiotemporal features include time dimension features and spatial dimension features; S24: Apply the missing pattern recognition rules in the supporting rules to identify missing samples and corresponding missing patterns in the real-time dimensionless data, label the current missing pattern of each missing sample, and form a real-time sample set to be completed.

[0013] As described above, the sub-steps for matching the current missing pattern of each missing sample in the real-time sample set to be completed with the corresponding completion model as the current completion model, inputting the real-time sample set to be completed into the current completion model, and outputting the completed data from the current completion model are as follows: S31: Read the current missing pattern of each missing sample in the real-time sample set to be completed, and use the completion model whose missing pattern matches the current missing pattern as the current completion model for the corresponding missing sample. Each core water quality parameter in each missing sample corresponds to a dedicated current completion model; S32: Input the real-time multi-source spatiotemporal features of the real-time sample set to be completed into the corresponding current completion model, and calculate the initial completed data for each core water quality parameter through the weighted fusion strategy built into the current completion model; S33: Perform inverse normalization processing on the initial completed data to obtain the completed data, and output it.

[0014] As shown above, the expression for obtaining the completed data by performing inverse normalization on the initial completed data is: ;in, To complete the data, the first... The parameters of the core water quality parameters after reverse normalization; For the first in real-time multi-parameter time series data The normalized values ​​of the core water quality parameters; For the first in real-time multi-parameter time series data The minimum value of the core water quality parameter after normalization; For the first in real-time multi-parameter time series data The maximum value of the core water quality parameter after normalization.

[0015] As described above, the sub-steps for obtaining the true value used to verify the completed sample, calculating the completion error between the completed data and the true value, analyzing the completion error using a tiered threshold method, determining the credibility level of the completed data, and outputting an evaluation report containing the completed data, completion error, and credibility level are as follows: S41: Obtain the true value used to verify the completed data, where the true value includes: the true value recovered by sensors after real-time data loss and the true value obtained by manual sampling detection; S42: Calculate the completion error between the completed data and the true value, where the completion error includes: absolute error and relative error; S43: Calculate the comprehensive credibility score based on the completion error; where the expression for the comprehensive credibility score is: ; in, For the first The overall credibility score of the core water quality parameters; For the first Error weights for key water quality parameters; For the first Spatial consistency weights of key water quality parameters; For the first Time consistency weights for key water quality parameters; ; For the first The relative error of the core water quality parameters; For the first The maximum permissible relative error of the preset core water quality parameters; For the first Spatial consistency of key water quality parameters; For the first Time consistency of core water quality parameters; S44: Analyze the overall reliability score using a graded threshold method to determine the reliability level of each core water quality parameter; when ,and At that time, the first The credibility level of the core water quality parameters is high; when ,and At that time, the first The reliability level of the core water quality parameters is medium confidence; when ,and At that time, the first The credibility level of the core water quality parameters is low; S45: Output an assessment report containing the supplementary data, supplementary error, comprehensive credibility score and credibility level of each core water quality parameter. The assessment report supports seamless integration with the environmental monitoring platform.

[0016] As described above, the sub-steps for incrementally training the completion model using newly added historical complete data according to preset update conditions to obtain the updated completion model are as follows: S51: Preset update conditions: when the amount of newly added historical complete data reaches 20% of the amount of data used in the previous training, or when the completion model outputs evaluation reports containing low confidence for three consecutive times, the completion model is triggered to update; S52: In response to the triggered update conditions, the completion model is incrementally trained using the dataset obtained by merging the newly added historical complete data with the previous training data, optimizing the parameters of the completion model to adapt to the new data distribution, and obtaining the updated initial completion model after verification after training; S53: The accuracy of the updated initial completion model is evaluated using the incremental validation set obtained by splitting the newly added historical complete data according to a preset ratio. If the average relative error of each core water quality parameter of the updated initial completion model is lower than that of the previous completion model, and the reduction reaches or exceeds the accuracy improvement threshold, the verification is passed, and the updated initial completion model is used as the updated completion model; if the verification fails, historical complete data continues to be accumulated and training and verification are repeated until the accuracy reaches the standard.

[0017] As shown above, for watersheds with small water quality fluctuations, the accuracy improvement threshold is 5%; for watersheds with complex water quality and varied missing patterns, the accuracy improvement threshold is 10%.

[0018] The beneficial effects achieved by this application are as follows: (1) The machine learning-based surface water monitoring data completion and credibility assessment method of this application can adapt to different missing patterns and integrate learning frameworks to achieve high-precision completion of surface water monitoring data by combining multi-source spatiotemporal features, and significantly reduce completion error.

[0019] (2) The machine learning-based surface water monitoring data completion and credibility assessment method of this application can construct a multi-dimensional assessment system of error, spatial consistency and temporal consistency, quantify the credibility level of the completed data, and realize the reliable assessment of the completed data.

[0020] (3) The machine learning-based surface water monitoring data completion and credibility assessment method of this application can achieve dynamic optimization of the completion model through incremental training, and adapt to various monitoring scenarios, effectively improving the data quality and application reliability of the surface water monitoring network. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0022] Figure 1 A flowchart of one embodiment of a machine learning-based method for completing and assessing the credibility of surface water monitoring data. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] like Figure 1 As shown, this application provides a method for surface water monitoring data completion and reliability assessment based on machine learning, including the following steps: S1: Acquire historical multi-parameter time-series data from the surface water monitoring network. Perform preprocessing, missing pattern identification, multi-source spatiotemporal feature extraction and filtering on the historical multi-parameter time-series data to obtain filtered features. Configure corresponding ensemble learning frameworks for three types of missing patterns: random missing, continuous missing, and systematic missing. Train and validate the ensemble learning frameworks based on the filtered features to obtain the completion model and supporting rules. The supporting rules include: preprocessing rules, feature extraction rules, and missing pattern identification rules.

[0025] Furthermore, historical multi-parameter time-series data from the surface water monitoring network are acquired. The historical multi-parameter time-series data undergoes preprocessing, missing pattern identification, multi-source spatiotemporal feature extraction, and filtering sequentially. The sub-steps for obtaining the filtered features are as follows: S11: Acquire historical multi-parameter time-series data from the surface water monitoring network, preprocess the historical multi-parameter time-series data, and convert the historical multi-parameter time-series data into dimensionless data in the [0,1] interval; wherein, the historical multi-parameter time-series data includes at least: pH value, dissolved oxygen (DO), turbidity, chemical oxygen demand (COD), and ammonia nitrogen (… Time-series data of eight core water quality parameters, including total phosphorus (TP), total nitrogen (TN), and permanganate index (CODMn).

[0026] Furthermore, a linear normalization formula is used to convert historical multi-parameter time series data into dimensionless data in the [0,1] interval. The expression for the linear normalization formula is as follows: ; in, The first in historical multi-parameter time series data The normalized values ​​of the core water quality parameters; The first in historical multi-parameter time series data The original parameter values ​​of the core water quality parameters; The first in historical multi-parameter time series data The minimum value of the core water quality parameter; The first in historical multi-parameter time series data The maximum value of the core water quality parameter.

[0027] Specifically, when historical multi-parameter time-series data includes time-series data for eight core water quality parameters, These correspond to pH value, dissolved oxygen (DO), turbidity, chemical oxygen demand (COD), and ammonia nitrogen, respectively. Total phosphorus (TP), total nitrogen (TN), and permanganate index (CODMn).

[0028] S12: Identify historical missing patterns based on the duration of missing data, distribution characteristics, and data correlation of dimensionless data; where historical missing patterns include any one or more of random missing, continuous missing, and systematic missing.

[0029] Specifically, random missing data refers to data missing at a single point in time, while adjacent time series data are complete, and the probability of missing data is unrelated to the data itself.

[0030] Continuous missing data refers to data missing for 6-72 consecutive hours, with no valid data records within the missing period.

[0031] Systematic data loss refers to data loss for more than 72 consecutive hours, which is mostly caused by systemic problems such as sensor failure.

[0032] S13: Extract multi-source spatiotemporal features from dimensionless data to obtain historical multi-source spatiotemporal features; among which, historical multi-source spatiotemporal features include time dimension features and spatial dimension features.

[0033] Among them, the time dimension features are extracted based on the time-series distribution patterns of dimensionless data, including periodic features, sliding time window trend features, and multi-scale lag features.

[0034] Specifically, periodicity refers to the recurring patterns of water quality parameters over time, including hourly cycles (such as concentration fluctuations at different times of the day), daily cycles (such as numerical trends at the same time over multiple consecutive days), and seasonal cycles (such as differences in water quality parameter changes in spring, summer, autumn, and winter). Sliding time window trend characteristics refer to setting time windows based on actual monitoring frequency (e.g., monitoring once per hour, then the time window contains 6-24 data points), calculating the mean (average level of dimensionless data within the time window), variance (dispersion of dimensionless data within the time window), and slope (rate of change of dimensionless data within the time window) of the dimensionless data within each time window to capture short-term time-series trends. Multi-scale lag characteristics refer to selecting dimensionless monitoring values ​​from 1 hour, 3 hours, 6 hours, and 12 hours before the completion time as lag characteristics to reflect the impact of historical data on the current completion value.

[0035] Among them, spatial dimension features are extracted based on the spatial correlation between the target monitoring station and surrounding stations, including topological correlation features and water quality similarity features.

[0036] Specifically, topological correlation features refer to the straight-line distance (in meters) between the target station and upstream and downstream stations and the direction of water flow (such as the flow direction from the upstream station to the target station and the flow direction from the target station to the downstream station), which are used to reflect the impact of hydrological topology on water quality; water quality similarity features refer to the correlation between the target station and the dimensionless data of 1-3 surrounding reference stations during the same period, calculated by Pearson correlation coefficient. The closer the coefficient is to 1, the more consistent the trend of water quality parameter changes between the two stations, thereby quantifying spatial similarity.

[0037] S14: The Recursive Feature Elimination (RFE) algorithm is used in conjunction with feature importance ranking to filter historical multi-source spatiotemporal features and obtain historical multi-source spatiotemporal features with cumulative contribution exceeding the contribution threshold as the filtered features.

[0038] Specifically, the selected features are used for subsequent model training.

[0039] Furthermore, the Recursive Feature Elimination (RFE) algorithm, combined with feature importance ranking, is used to filter historical multi-source spatiotemporal features. The sub-steps for obtaining historical multi-source spatiotemporal features with a cumulative contribution exceeding the contribution threshold as the filtered features are as follows: S141: Construct an initial contribution analysis model based on historical multi-source spatiotemporal features. Train the initial contribution analysis model using complete water quality data selected from historical multi-parameter time series data to obtain the contribution analysis model. Quantify the contribution of each historical multi-source spatiotemporal feature to the water quality parameter prediction results through the feature importance evaluation mechanism built into the contribution analysis model. Generate a feature importance ranking table from high to low contribution, thus completing the construction of the feature importance ranking.

[0040] Specifically, the complete water quality data selected from historical multi-parameter time-series data consists of time-series data in which no core water quality parameter is missing.

[0041] Furthermore, the specific type of contribution analysis model is set according to actual needs. In this application, the preferred model is the XGBoost model, and the corresponding feature importance evaluation mechanism is the information gain evaluation of XGBoost.

[0042] S142: Use the features in the feature importance ranking table as the set to be filtered and set the contribution threshold; use the set to be filtered and the contribution threshold as the initialized filtering parameters.

[0043] Specifically, the contribution threshold value is set according to actual needs, and in this application, it is preferably 80%.

[0044] S143: Select the first historical multi-source spatiotemporal feature that is not in the added feature list from the set to be screened in ascending order of feature importance ranking table, add the historical multi-source spatiotemporal feature to the temporary feature subset, and simultaneously add the historical multi-source spatiotemporal feature to the added feature list. Then, use the updated temporary feature subset as the current temporary feature subset to input the contribution analysis model for training, and calculate the cumulative contribution of the current temporary feature subset.

[0045] Specifically, the ascending order of the feature importance ranking table refers to the order from the highest to the lowest contribution.

[0046] Furthermore, the expression for the cumulative contribution of the current temporary feature subset is: ; in, The cumulative contribution represents the proportion of the total contribution of the current temporary feature subset to the total contribution of all historical multi-source spatiotemporal features. For the current temporary feature subset, the first The individual contribution of a historical multi-source spatiotemporal feature , This represents the total number of historical multi-source spatiotemporal features in the current temporary feature subset; The 1st in order of feature importance The individual contribution of a historical multi-source spatiotemporal feature , The total number of historical multi-source spatiotemporal features; The total contribution of all historical multi-source spatiotemporal features.

[0047] Specifically, the first in the current temporary feature subset Individual contribution of historical multi-source spatiotemporal features The contribution of historical multi-source spatiotemporal features added sequentially from the set to be screened after being sorted by feature importance. The information gain was determined by the contribution analysis model. The information gain was determined by the contribution analysis model. It is a fixed value.

[0048] S144: Determine whether the cumulative contribution has reached the contribution threshold. If not, continue to select the next historical multi-source spatiotemporal feature not in the list of added features from the set to be filtered in ascending order of feature importance ranking table, and repeat S143. If it has reached the threshold, immediately stop adding historical multi-source spatiotemporal features, use the current temporary feature subset as the filtered features, and discard the remaining unadded historical multi-source spatiotemporal features in the set to be filtered.

[0049] Furthermore, for the three missing data patterns—random missing data, continuous missing data, and systematic missing data—corresponding ensemble learning frameworks are configured. The ensemble learning frameworks are trained and validated based on the selected features to obtain the completion model and its accompanying rules. The sub-steps are as follows: S11': Configure the corresponding ensemble learning frameworks for the three missing data patterns: random missing data, continuous missing data, and systematic missing data.

[0050] As an example, the random forest framework is used as the ensemble learning framework for random missing values; the gradient boosting tree framework is used as the ensemble learning framework for continuous missing values; and the support vector machine or logistic regression framework is used as the ensemble learning framework for systematic missing values.

[0051] Specifically, the type of ensemble learning framework for random missing data is selected based on actual needs. This application preferably uses a Random Forest framework, which leverages the ensemble prediction capabilities of multiple decision trees to reduce the overfitting risk of individual trees through voting or averaging, thereby enhancing robustness to randomly missing data. The type of ensemble learning framework for continuous missing data is also selected based on actual needs. This application preferably uses a Gradient Boosting Tree (e.g., XGBoost / LightGBM) framework, which iteratively trains weak classifiers and accumulates weights to continuously optimize the loss function, accurately adapting to the time-series dependency characteristics of continuously missing data. The type of ensemble learning framework for systematic missing data is also selected based on actual needs. This application preferably uses a Support Vector Machine (SVM) or Logistic Regression framework, which utilizes high-dimensional space mapping or linear classification capabilities to achieve accurate matching and completion of systematic missing patterns, thus completing the ensemble learning framework configuration for the three types of missing data.

[0052] As another example, all three missing modes employ an ensemble learning framework consisting of a main model and an auxiliary model, and dynamically adjust the model weights for different missing modes: An ensemble learning framework for random missing data (RM): It primarily uses a lightweight XGBoost model (weight 0.7) and secondarily uses an LSTM model (weight 0.3). It leverages the efficiency of XGBoost to capture nonlinear relationships in features, while supplementing time series characteristics through LSTM to improve the robustness of completion in random missing data scenarios.

[0053] An ensemble learning framework for continuous missing data (CM): It mainly uses a bidirectional LSTM (Bi-LSTM) model (weight 0.6) and a spatiotemporal attention mechanism (STA) (weight 0.4). Bi-LSTM captures long-term temporal dependencies, while STA strengthens key spatiotemporal features, accurately adapting to the temporal series dependency characteristics of continuous missing data.

[0054] An ensemble learning framework for systematic missing (SM) models: A graph neural network (GNN) is used in fusion with an XGBoost model (weight ratio of 0.5:0.5). The GNN is used to model the spatial relationships between sites, while the efficient computing power of XGBoost is used to solve the problem of historical dependency breaks in long-term missing data, so as to achieve accurate matching and completion of systematic missing patterns.

[0055] S12': Based on the selected features, the ensemble learning framework for the three missing patterns is trained and validated to obtain the completion model for the three missing patterns.

[0056] Specifically, as one example, the selected features are used as input, and the dataset is divided into random missing, consecutive missing, and systematic missing categories. The training and validation sets are split in a 7:3 ratio. For the dataset with random missing features, a random forest framework is used to train its corresponding ensemble learning framework, resulting in a trained model with random missing features. During training, the tree depth of the random forest framework is set to 5-10, and the subsampling ratio is set to 0.7-0.9. For the dataset with consecutive missing features, a gradient boosting tree framework is used to train its corresponding ensemble learning framework, resulting in a trained model with consecutive missing features. During training, the mean squared error (MSE) is used as the loss function, and the learning rate of the gradient boosting tree framework is set to 0. Set the iteration count to 100-500. For the systematically missing dataset, train the corresponding ensemble learning framework using SVM or logistic regression to obtain the trained model with systematic missing data. Set the regularization parameter during training (e.g., C value of SVM is 0.1-10). Validate the trained model with continuous missing data using mean squared error (MSE) and coefficient of determination (R²) to obtain the completed model with continuous missing data. Validate the trained models with random missing data and systematically missing data using accuracy, F1 score, and confusion matrix respectively to obtain the completed models with random missing data and systematically missing data, thus completing the training and validation of the completed models.

[0057] As another embodiment, the Huber loss function is used to train and validate the ensemble learning framework consisting of the main model and the auxiliary model to obtain a complete model for three missing modes. The choice of loss function is not limited to the Huber loss function. The loss function can be the existing Huber loss function, or other existing loss functions adapted to this ensemble learning framework (such as MAE, cross-entropy, etc.).

[0058] S13': The following preprocessing rules will be used: 3σ principle (three standard deviations principle) or IQR method (interquartile range method) to handle outliers, standardize multi-dimensional features, and unify time series timestamps to achieve time alignment; the following feature extraction rules will be used: extracting spatial features based on monitoring station locations, extracting temporal features based on collection time, calculating statistical features, and constructing feature interaction terms; the following missing value pattern recognition rules will be used: identifying random missing values ​​through missing value distribution analysis and statistical tests, identifying continuous missing values ​​based on time series analysis, and identifying systematic missing values ​​with the help of pattern matching and cluster analysis; the preprocessing rules, feature extraction rules, and missing value pattern recognition rules will be used together as supporting rules.

[0059] S2: Acquire real-time multi-parameter time-series data of the surface water monitoring network, apply supporting rules to preprocess the real-time multi-parameter time-series data, extract multi-source spatiotemporal features and identify missing patterns, and label the current missing pattern of each missing sample to form a real-time sample set to be completed.

[0060] Furthermore, the following sub-steps are taken to obtain real-time multi-parameter time-series data from the surface water monitoring network, apply supporting rules to preprocess the real-time multi-parameter time-series data, extract multi-source spatiotemporal features and identify missing patterns, and label the current missing pattern of each missing sample to form a real-time sample set to be completed: S21: Acquire real-time multi-parameter time-series data from the surface water monitoring network, wherein the real-time multi-parameter time-series data shall include at least: pH value, dissolved oxygen (DO), turbidity, chemical oxygen demand (COD), and ammonia nitrogen ( ). Time-series data of eight core water quality parameters, including total phosphorus (TP), total nitrogen (TN), and permanganate index (CODMn).

[0061] Specifically, the parameter types of real-time multi-parameter time series data are consistent with those of historical multi-parameter time series data.

[0062] S22: Apply the preprocessing rules in the supporting rules to preprocess the real-time multi-parameter time series data to obtain real-time dimensionless data.

[0063] S23: Apply the feature extraction rules in the supporting rules to extract multi-source spatiotemporal features from real-time dimensionless data to obtain real-time multi-source spatiotemporal features; among them, real-time multi-source spatiotemporal features include time dimension features and spatial dimension features.

[0064] S24: Apply the missing pattern recognition rules in the supporting rules to identify missing samples and corresponding missing patterns in real-time dimensionless data, label the current missing pattern of each missing sample, and form a real-time sample set to be completed.

[0065] Specifically, the real-time incomplete sample set includes real-time missing data for various core water quality parameters, corresponding spatiotemporal features, and missing pattern labels.

[0066] S3: Match the corresponding completion model to the current completion model based on the current missing pattern of each missing sample in the real-time sample set to be completed. Input the real-time sample set to be completed into the current completion model and output the completed data from the current completion model.

[0067] Furthermore, the following sub-steps are taken: matching the corresponding completion model based on the current missing pattern of each missing sample in the real-time imputation sample set as the current completion model; inputting the real-time imputation sample set into the current completion model; and having the current completion model output the completed data: S31: Read the current missing pattern of each missing sample in the real-time sample set to be completed, and use the completion model that matches the current missing pattern as the current completion model of the corresponding missing sample. Each core water quality parameter in each missing sample corresponds to a unique current completion model.

[0068] Specifically, the supplementary models are configured independently according to the core water quality parameters, that is, each core water quality parameter corresponds to a dedicated current supplementary model, in order to ensure the correctness and accuracy of the supplementation.

[0069] S32: Input the real-time multi-source spatiotemporal features of the real-time sample set to be completed into the corresponding current completion model, and calculate the initial completion data of each core water quality parameter through the weighted fusion strategy built into the current completion model.

[0070] Specifically, step S32 can be implemented using existing technology, so it will not be described in detail here.

[0071] S33: Perform inverse normalization on the initial completed data to obtain the completed data, and output it.

[0072] Specifically, the scale of the completed data obtained by inverse normalization of the initial completed data is consistent with the scale of the original data.

[0073] Furthermore, the initial completed data is denormalized to obtain the expression for the completed data: ; in, To complete the data, the first... The parameters of the core water quality parameters after reverse normalization; For the first in real-time multi-parameter time series data The normalized values ​​of the core water quality parameters; For the first in real-time multi-parameter time series data The minimum value of the core water quality parameter after normalization; For the first in real-time multi-parameter time series data The maximum value of the core water quality parameter after normalization.

[0074] S4: Obtain the true value used to verify the completed data, calculate the completion error between the completed data and the true value, analyze the completion error using the graded threshold method, determine the credibility level of the completed data, and output an evaluation report containing the completed data, completion error, and credibility level.

[0075] Furthermore, the sub-steps for obtaining the true values ​​used to verify the completed samples, calculating the completion error between the completed data and the true values, analyzing the completion error using a tiered threshold method, determining the credibility level of the completed data, and outputting an evaluation report containing the completed data, completion error, and credibility level are as follows: S41: Obtain the true value used to verify the completed data, wherein the true value includes: the true value obtained by sensor recovery after real-time data is missing and the true value obtained by manual sampling detection.

[0076] Specifically, the true values ​​obtained through sensor recovery after real-time data loss are: the measured data of each core water quality parameter collected at the corresponding completion point after the data loss period ends and the sensor resumes normal operation. The true values ​​obtained through manual sampling and testing are: the measured data of each core water quality parameter obtained through manual on-site sampling and laboratory testing for the data loss period. This application adopts dual-source verification of sensor-recovered values ​​and manual sampling values, breaking through the single true value acquisition mode, ensuring the comprehensiveness, relevance, and completeness of the true values, providing high-quality data support for error calculation and evaluation of each core water quality parameter, and significantly reducing verification bias.

[0077] S42: Calculate the completion error between the completed data and the true value, where the completion error includes: absolute error and relative error.

[0078] Furthermore, the expression for the absolute error is: ; The expression for relative error is: ; in, For the first The absolute error of the core water quality parameters; For the first The relative error of the core water quality parameters; For the first Complete data for key water quality parameters; For the first The verification true value of each core water quality parameter is determined as follows: when the true value recovered by the sensor exists and the data is valid (i.e., the true value recovered by the sensor after the real-time data is missing is within the normal range of the core water quality parameter and has no obvious abnormal fluctuations), the true value recovered by the sensor after the real-time data is missing is preferentially selected as the verification true value; when the sensor cannot recover the data or the recovered data is invalid, the true value obtained by manual sampling and detection is selected as the verification true value, ensuring that each supplementary data has a unique corresponding valid true value for error calculation.

[0079] This application can comprehensively characterize the completion accuracy through dual errors (absolute error reflects numerical deviation, and relative error reflects proportional deviation), avoiding the limitations of a single error.

[0080] S43: Calculate the overall credibility score based on the completion error.

[0081] Furthermore, the expression for the overall credibility score is as follows: ; in, For the first The overall credibility score of the core water quality parameters; For the first Error weights for key water quality parameters; For the first Spatial consistency weights of key water quality parameters; For the first Time consistency weights for key water quality parameters; ; For the first The relative error of the core water quality parameters; For the first The maximum permissible relative error of the preset core water quality parameters; For the first Spatial consistency of key water quality parameters; For the first The time consistency of the core water quality parameters.

[0082] Specifically, The value range is [0,1]. The specific value is set according to actual needs, and the preferred value in this application is 0.6. The specific value is set according to actual needs, and the preferred value in this application is 0.2. The specific value is set according to actual needs, and the preferred value in this application is 0.2. Take 20%. Through the first The correlation between the supplementary data of the core water quality parameters and the data of the same period at surrounding stations was obtained through calculation. , The value range is [0,1]. Through the first... The trend fit between the supplementary data of the core water quality parameters and the preceding and following time series data was calculated. , The value range is [0,1].

[0083] The expression for the overall credibility score in this application constructs a multi-dimensional evaluation system by integrating error accuracy, spatial consistency, and temporal consistency. This system not only conforms to the spatiotemporal variation patterns of water quality data but also comprehensively verifies the reliability of the supplementary data from multiple dimensions, further improving the accuracy and credibility of the supplementary data and providing high-quality data support for subsequent model updates and water quality analysis.

[0084] S44: The reliability score is analyzed using a tiered threshold method to determine the reliability level of each core water quality parameter; when ,and At that time, the first The credibility level of the core water quality parameters is high; when ,and At that time, the first The reliability level of the core water quality parameters is medium confidence; when ,and At that time, the first The credibility level of the core water quality parameters is low.

[0085] Specifically, when the first When the confidence level of a core water quality parameter is low, manual verification or supplementary sampling is required.

[0086] S45: Outputs an assessment report containing complete data for each core water quality parameter, completion error, overall credibility score, and credibility level. The assessment report supports seamless integration with the environmental monitoring platform.

[0087] Specifically, step S4 achieves a comprehensive and accurate assessment of the credibility of the supplementary data through multivariate true value verification, dual error calculation, spatiotemporal consistency fusion, and refined classification. This effectively improves the objectivity and practicality of the assessment report and provides a reliable guarantee for the subsequent application of surface water monitoring data.

[0088] S5: According to the preset update conditions, the newly added historical complete data is used to incrementally train the completion model to obtain the updated completion model.

[0089] Furthermore, the sub-steps for incrementally training the completion model using newly added complete historical data according to preset update conditions to obtain the updated completion model are as follows: S51: Preset update conditions: when the amount of newly added historical complete data reaches 20% of the amount of data used in the last training, or when the complete model outputs evaluation reports with low confidence for 3 consecutive times, the complete model update is triggered.

[0090] Specifically, the newly added historical complete data volume refers to the newly accumulated, complete, and unmissing historical multi-parameter time series data since the last model completion update.

[0091] The amount of data used in the last training session refers to the total size of the complete historical data used when the model was last updated and trained. The total size of the complete historical data includes all the complete historical data accumulated before the last update, as well as the complete data added and incorporated into the training before the last update. It does not only refer to the amount of data used in the initial training, but rather to the total amount of training data dynamically accumulated with each update.

[0092] S52: In response to the triggered update condition, the newly added historical complete data and the dataset merged with the previous training data are used to incrementally train the completion model, optimize the parameters of the completion model to adapt to the new data distribution, and obtain the updated initial completion model after the training is completed and verified.

[0093] S53: The accuracy of the updated initial complete model is evaluated by the incremental validation set obtained by splitting the newly added historical complete data according to a preset ratio. If the average relative error of each core water quality parameter of the updated initial complete model is lower than that of the previous complete model, and the reduction reaches or exceeds the accuracy improvement threshold, the validation is passed, and the updated initial complete model is used as the updated complete model; if the validation fails, historical complete data is accumulated and training and validation are repeated until the accuracy reaches the standard.

[0094] Specifically, the value of the preset ratio is set according to actual needs, and the preferred ratio in this application is 3:7.

[0095] The accuracy improvement threshold is a single set value, which can be dynamically adjusted according to actual monitoring needs and the complexity of watershed water quality. The value range is 5% to 10% (the upper limit is set at 10% to avoid overfitting due to overtraining; the lower limit is set at 5% to ensure that the model update has practical application value). For example, for watersheds with small water quality fluctuations, the accuracy improvement threshold can be set to 5%, meaning that a reduction in the average relative error of 5% or more is considered a successful validation. For watersheds with complex water quality and varied missing patterns, the accuracy improvement threshold can be set to 10%, meaning that a reduction in the average relative error of 10% or more is considered a successful validation.

[0096] Furthermore, after step S52, step S52' is also included: calculating the incremental update efficiency of the updated initial complete model by updating the evaluation model; if the incremental update efficiency is greater than or equal to the preset model update efficiency threshold, then S53 is executed; if the incremental update efficiency is less than the preset model update efficiency threshold, then the current update process is paused, and the newly added historical complete data continues to accumulate until the ratio of the amount of newly added historical complete data to the amount of the previous training data reaches the preset ratio threshold, then S52 and S52' are executed again.

[0097] Specifically, the preset model update efficiency threshold is set according to actual needs, and is preferably 0.05 in this application. The preset ratio threshold is set according to actual needs, and is preferably 30% in this application.

[0098] Furthermore, the expression for calculating the incremental update efficiency of the updated initial completed model by updating the evaluation model is as follows: ; in, To improve the efficiency of incremental model updates; This represents the accuracy improvement of the updated initial completed model compared to the original completed model, where... =The average relative error of the original completed model - the average relative error of the updated initial completed model; The time taken for this incremental training (in hours); This is the ratio of the amount of newly added historical complete data to the amount of the previous training data (e.g., if the addition is 20%). ).

[0099] Specifically, by adding step S52', the quantification formula can be used. Accurately assess the cost-effectiveness of incremental model updates, avoid resource waste caused by blind training, and make the decision to update the model more scientific and efficient.

[0100] The beneficial effects achieved by this application are as follows: (1) The machine learning-based surface water monitoring data completion and credibility assessment method of this application can adapt to different missing patterns and integrate learning frameworks to achieve high-precision completion of surface water monitoring data by combining multi-source spatiotemporal features, and significantly reduce completion error.

[0101] (2) The machine learning-based surface water monitoring data completion and credibility assessment method of this application can construct a multi-dimensional assessment system of error, spatial consistency and temporal consistency, quantify the credibility level of the completed data, and realize the reliable assessment of the completed data.

[0102] (3) The machine learning-based surface water monitoring data completion and credibility assessment method of this application can achieve dynamic optimization of the completion model through incremental training, and adapt to various monitoring scenarios, effectively improving the data quality and application reliability of the surface water monitoring network.

[0103] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the scope of protection of this application is intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application. Obviously, those skilled in the art can make various alterations and variations to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of protection of this application and its equivalents, this application also intends to include these modifications and variations.

Claims

1. A method for completing and assessing the reliability of surface water monitoring data based on machine learning, characterized in that, Includes the following steps: S1: Acquire historical multi-parameter time-series data of the surface water monitoring network, and perform preprocessing, missing pattern identification, multi-source spatiotemporal feature extraction and filtering on the historical multi-parameter time-series data in sequence to obtain the filtered features; For three types of missing data patterns—random missing data, continuous missing data, and systematic missing data—corresponding ensemble learning frameworks are configured. The ensemble learning frameworks are trained and validated based on the selected features to obtain the completion model and supporting rules. The supporting rules include: preprocessing rules, feature extraction rules, and missing data pattern recognition rules. S2: Acquire real-time multi-parameter time-series data of the surface water monitoring network, apply supporting rules to preprocess the real-time multi-parameter time-series data, extract multi-source spatiotemporal features and identify missing patterns, and label the current missing pattern of each missing sample to form a real-time sample set to be completed. S3: Match the corresponding completion model based on the current missing pattern of each missing sample in the real-time sample set to be completed as the current completion model, input the real-time sample set to be completed into the current completion model, and output the completed data from the current completion model; S4: Obtain the true value used to verify the completed data, calculate the completion error between the completed data and the true value, analyze the completion error using the graded threshold method, determine the credibility level of the completed data, and output an evaluation report containing the completed data, completion error, and credibility level. S5: According to the preset update conditions, the newly added historical complete data is used to incrementally train the completion model to obtain the updated completion model; All three missing data types employ an ensemble learning framework consisting of a main model and an auxiliary model, with model weights dynamically adjusted for different missing data types. An ensemble learning framework with random missing values: a lightweight XGBoost model as the master model with a weight of 0.7, and an LSTM model as the auxiliary model with a weight of 0.

3. The continuous missing ensemble learning framework uses a bidirectional LSTM model as the main model with a weight of 0.6, and a spatiotemporal attention mechanism as the auxiliary model with a weight of 0.

4. An integrated learning framework for addressing systemic deficiencies: A graph neural network is used in fusion with an XGBoost model, with a weight ratio of 0.5:0.5 between the graph neural network and the XGBoost model.

2. The method for completing and assessing the reliability of surface water monitoring data based on machine learning according to claim 1, characterized in that, The following are the sub-steps for acquiring historical multi-parameter time-series data from the surface water monitoring network, followed by preprocessing, missing pattern identification, multi-source spatiotemporal feature extraction and filtering, and obtaining the filtered features: S11: Acquire historical multi-parameter time-series data from the surface water monitoring network, preprocess the historical multi-parameter time-series data, and convert the historical multi-parameter time-series data into dimensionless data in the [0,1] interval; wherein, the historical multi-parameter time-series data shall include at least the time-series data of the following eight core water quality parameters: pH value, dissolved oxygen, turbidity, chemical oxygen demand, ammonia nitrogen, total phosphorus, total nitrogen and permanganate index; S12: Identify historical missing patterns based on the duration of missing data, distribution characteristics, and data correlation of dimensionless data; where historical missing patterns include any one or more of random missing, continuous missing, and systematic missing. S13: Extract multi-source spatiotemporal features from dimensionless data to obtain historical multi-source spatiotemporal features; among which, historical multi-source spatiotemporal features include time dimension features and spatial dimension features; S14: A recursive feature elimination algorithm is adopted, combined with feature importance ranking, to filter historical multi-source spatiotemporal features and obtain historical multi-source spatiotemporal features with cumulative contribution exceeding the contribution threshold as the filtered features.

3. The method for completing and assessing the reliability of surface water monitoring data based on machine learning according to claim 2, characterized in that, The following sub-steps employ a recursive feature elimination algorithm, combined with feature importance ranking, to filter historical multi-source spatiotemporal features and obtain those with a cumulative contribution exceeding a contribution threshold as the filtered features: S141: Construct an initial contribution analysis model based on historical multi-source spatiotemporal features. Train the initial contribution analysis model using complete water quality data selected from historical multi-parameter time series data to obtain the contribution analysis model. Quantify the contribution of each historical multi-source spatiotemporal feature to the water quality parameter prediction results through the feature importance evaluation mechanism built into the contribution analysis model. Generate a feature importance ranking table from high to low contribution, thus completing the construction of the feature importance ranking. S142: Use the features in the feature importance ranking table as the set to be filtered and set the contribution threshold; use the set to be filtered and the contribution threshold as the initialized filtering parameters; S143: Select the first historical multi-source spatiotemporal feature that is not in the added feature list from the set to be screened in ascending order of feature importance ranking table, add the historical multi-source spatiotemporal feature to the temporary feature subset, and simultaneously add the historical multi-source spatiotemporal feature to the added feature list. Then, use the updated temporary feature subset as the current temporary feature subset to input the contribution analysis model for training, and calculate the cumulative contribution of the current temporary feature subset. S144: Determine whether the cumulative contribution has reached the contribution threshold. If not, continue to select the next historical multi-source spatiotemporal feature not in the list of added features from the set to be filtered in ascending order of feature importance ranking table, and repeat S143. If it has reached the threshold, immediately stop adding historical multi-source spatiotemporal features, use the current temporary feature subset as the filtered features, and discard the remaining unadded historical multi-source spatiotemporal features in the set to be filtered.

4. The method for completing and assessing the reliability of surface water monitoring data based on machine learning according to claim 1, characterized in that, For three types of missing data—random missing data, continuous missing data, and systematic missing data—corresponding ensemble learning frameworks are configured. The ensemble learning frameworks are trained and validated based on the selected features, resulting in the following sub-steps for obtaining the completion model and accompanying rules: S11': Configure the corresponding ensemble learning frameworks for the three missing data patterns: random missing data, continuous missing data, and systematic missing data. S12': Based on the selected features, the ensemble learning framework for the three missing patterns is trained and validated to obtain the completion model for the three missing patterns; S13': The following preprocessing rules will be used: 3σ principle or IQR method to handle outliers, standardize multi-dimensional features, and unify time series timestamps to achieve time alignment; the following feature extraction rules will be used: extracting spatial features based on monitoring station locations, extracting temporal features based on collection time, calculating statistical features, and constructing feature interaction terms; the following missing value pattern recognition rules will be used: identifying random missing values ​​through missing value distribution analysis and statistical tests, identifying continuous missing values ​​based on time series analysis, and identifying systematic missing values ​​with the help of pattern matching and cluster analysis; the preprocessing rules, feature extraction rules, and missing value pattern recognition rules will be used together as supporting rules.

5. The method for completing and assessing the reliability of surface water monitoring data based on machine learning according to claim 1, characterized in that, The following are the sub-steps for acquiring real-time multi-parameter time-series data from the surface water monitoring network, applying supporting rules to preprocess the real-time multi-parameter time-series data, extracting multi-source spatiotemporal features and identifying missing patterns, and labeling the current missing pattern of each missing sample to form a real-time sample set to be completed: S21: Obtain real-time multi-parameter time-series data from the surface water monitoring network. The real-time multi-parameter time-series data shall include at least the time-series data of the following eight core water quality parameters: pH, dissolved oxygen, turbidity, chemical oxygen demand, ammonia nitrogen, total phosphorus, total nitrogen and permanganate index. S22: Apply the preprocessing rules in the supporting rules to preprocess the real-time multi-parameter time series data to obtain real-time dimensionless data; S23: Apply the feature extraction rules in the supporting rules to extract multi-source spatiotemporal features from real-time dimensionless data to obtain real-time multi-source spatiotemporal features; among which, real-time multi-source spatiotemporal features include time dimension features and spatial dimension features; S24: Apply the missing pattern recognition rules in the supporting rules to identify missing samples and corresponding missing patterns in real-time dimensionless data, label the current missing pattern of each missing sample, and form a real-time sample set to be completed.

6. The method for completing and assessing the reliability of surface water monitoring data based on machine learning according to claim 1, characterized in that, The following are the sub-steps for matching the current missing pattern of each missing sample in the real-time imputation sample set to the corresponding imputation model as the current imputation model, inputting the real-time imputation sample set into the current imputation model, and the current imputation model outputting the imputed data: S31: Read the current missing pattern of each missing sample in the real-time sample set to be completed, and take the completion model that matches the current missing pattern as the current completion model of the corresponding missing sample. Each core water quality parameter in each missing sample corresponds to a unique current completion model. S32: Input the real-time multi-source spatiotemporal features of the real-time sample set to be completed into the corresponding current completion model, and calculate the initial completion data of each core water quality parameter through the weighted fusion strategy built into the current completion model; S33: Perform inverse normalization on the initial completed data to obtain the completed data, and output it.

7. The method for completing and assessing the reliability of surface water monitoring data based on machine learning according to claim 6, characterized in that, The initial completed data is denormalized to obtain the expression for the completed data: ； in, To complete the data, the first... The parameters of the core water quality parameters after reverse normalization; For the first in real-time multi-parameter time series data The normalized values ​​of the core water quality parameters; For the first in real-time multi-parameter time series data The minimum value of the core water quality parameter after normalization; For the first in real-time multi-parameter time series data The maximum value of the core water quality parameter after normalization.

8. The method for completing and evaluating the reliability of surface water monitoring data based on machine learning according to claim 1, characterized in that, The sub-steps for obtaining the true values ​​used to validate the completed samples, calculating the completion error between the completed data and the true values, analyzing the completion error using a tiered threshold method, determining the confidence level of the completed data, and outputting an evaluation report containing the completed data, completion error, and confidence level are as follows: S41: Obtain the true value used to verify the completed data, wherein the true value includes: the true value obtained by sensor recovery after real-time data is missing and the true value obtained by manual sampling detection; S42: Calculate the completion error between the completed data and the true value, where the completion error includes: absolute error and relative error; S43: Calculate the overall credibility score based on the completion error; where the expression for the overall credibility score is: ； in, For the first The overall credibility score of the core water quality parameters; For the first Error weights for key water quality parameters; For the first Spatial consistency weights of key water quality parameters; For the first Time consistency weights for key water quality parameters; ; For the first The relative error of the core water quality parameters; For the first The maximum permissible relative error of the preset core water quality parameters; For the first Spatial consistency of key water quality parameters; For the first Time consistency of key water quality parameters; S44: The reliability score is analyzed using a tiered threshold method to determine the reliability level of each core water quality parameter; when ,and At that time, the first The credibility level of the core water quality parameters is high; when ,and At that time, the first The reliability level of the core water quality parameters is medium confidence; when ,and At that time, the first The credibility level of the core water quality parameters is low. S45: Outputs an assessment report containing complete data for each core water quality parameter, completion error, overall credibility score, and credibility level. The assessment report supports seamless integration with the environmental monitoring platform.

9. The method for completing and assessing the reliability of surface water monitoring data based on machine learning according to claim 1, characterized in that, The sub-steps for incrementally training the completion model using newly added complete historical data according to preset update conditions to obtain the updated completion model are as follows: S51: Preset update conditions: when the amount of newly added historical complete data reaches 20% of the amount of data used in the last training, or when the complete model outputs evaluation reports with low confidence for 3 consecutive times, the complete model is triggered to update. S52: In response to the triggered update condition, the newly added historical complete data and the dataset after merging with the previous training data are used to incrementally train the completion model, optimize the parameters of the completion model to adapt to the new data distribution, and obtain the updated initial completion model after the training is completed and verified. S53: The accuracy of the updated initial complete model is evaluated by the incremental validation set obtained by splitting the newly added historical complete data according to a preset ratio. If the average relative error of each core water quality parameter of the updated initial complete model is lower than that of the previous complete model, and the reduction reaches or exceeds the accuracy improvement threshold, the validation is passed, and the updated initial complete model is used as the updated complete model; if the validation fails, historical complete data is accumulated and training and validation are repeated until the accuracy reaches the standard.

10. The method for completing and assessing the reliability of surface water monitoring data based on machine learning according to claim 9, characterized in that, For watersheds with relatively small water quality fluctuations, the accuracy improvement threshold is 5%; for watersheds with complex water quality and varied missing patterns, the accuracy improvement threshold is 10%.

Images