Multi-periodic ecological environment index anomaly identification and deletion interpolation method

Abstract

The invention discloses a multi-periodic ecological environment index anomaly recognition and deletion interpolation method, which comprises the following steps of S1, performing time sequence decomposition on each ecological environment index of multi-periodic ecological environment data based on MSTL (Multiple Standard Template Library); s2, standardizing a residual list obtained by decomposition into a feature matrix, and inputting the feature matrix into a multivariable isolated forest model for abnormal value identification; s3, combining the abnormal value and the original missing value to obtain a time sequence containing the missing value, then adopting a Bayesian optimization algorithm to perform optimization on hyper-parameters of the MSTL, performing iterative search and determining an optimal hyper-parameter combination; s4, performing MSTL decomposition by using the optimal hyper-parameter, performing independent interpolation on multiple indexes by using respective decomposition results, and performing reconstruction to obtain a complete ecological environment index time sequence; the method supports multi-index parallel processing, abnormal values in the water quality data can be accurately identified, high-precision interpolation is performed by using data periodic characteristics, the quality of the water quality data is effectively improved, and the method has good popularization and application prospects.

Description

Technical Field

[0001] The present invention belongs to the technical field of water quality monitoring and management, and specifically relates to a method for anomaly identification and missing interpolation of multi-periodic ecological environment indicators. Background Art

[0002] Currently, high-frequency, multi-period automatic water quality monitoring technology, a key tool for water environment management, has been widely used in key river basins. The widespread establishment of automatic water quality monitoring stations provides a high-frequency, multi-period data foundation for identifying the dynamic patterns of water quality parameters. Its intensive monitoring capabilities, ranging from minute to hour levels, accurately capture the spatiotemporal evolution of physical and chemical indicators such as dissolved oxygen and water temperature, becoming a key support for precise pollution control in river basins. However, the continuous operation of the instruments is subject to environmental interference, resulting in widespread omissions, errors, and outliers in the data. Traditional manual verification methods are difficult to implement due to the large amount of data.

[0003] The unique characteristics of multi-period data exacerbate the difficulty of cleaning. First, automated monitoring instruments are susceptible to environmental interference during continuous operation, and missing segments often exhibit a non-random distribution. More importantly, such data exhibit strong autocorrelation and multi-period coupling, making it easy for conventional anomaly detection methods, such as the generalized extreme studentized deviation method, to misclassify natural fluctuations as anomalies. Second, the complex nonlinear relationships between variables cause traditional interpolation methods to produce results that violate the laws of materialization. While existing research has attempted to incorporate predictive models for real-time anomaly screening, these efforts focus on short-term forecasting rather than full-series cleaning. Furthermore, they fail to address the issues of period decoupling and embedding materialization mechanisms, making them difficult to meet the requirements for data authenticity, accuracy, and comprehensiveness. Therefore, there is an urgent need to develop anomaly identification and missing segment interpolation methods for batch cleaning that adapt to multi-period characteristics and integrate time series decomposition with mechanism constraints. Summary of the Invention

[0004] To solve the above problems, the present invention proposes a method for anomaly identification and missing interpolation of multi-periodic ecological environmental indicators. This method supports parallel processing of multiple indicators, can accurately identify outliers in water quality data, and use the periodic characteristics of the data for high-precision interpolation, effectively improving the quality of water quality data, and has good prospects for promotion and application.

[0005] To achieve the above object, the present invention adopts the following technical solutions: A method for identifying anomalies and interpolating missing values ​​of multi-periodic ecological and environmental indicators includes the following steps: S1. Based on MSTL, time series decomposition is performed on each ecological and environmental indicator of multi-periodic ecological and environmental data; S2, normalize the residual list obtained from the decomposition into a feature matrix and input it into the multivariate isolation forest model for outlier identification; S3. Merge outliers and original missing values ​​to obtain a time series with missing values. Then use the Bayesian optimization algorithm to optimize the hyperparameters of MSTL, iteratively search, and determine the optimal hyperparameter combination. S4. Use the optimal hyperparameters to perform MSTL decomposition, and independently interpolate multiple indicators using their respective decomposition results to reconstruct a complete time series of ecological environmental indicators.

[0006] Preferably, the specific process of step S1 is: S11. Input a list of multiple multi-periodic continuous ecological and environmental indicators, set the index to the timestamp type, and record the missing value locations of the original data; S12. Select the default MSTL hyperparameters and apply MSTL decomposition to each ecological environment indicator independently to obtain the decomposition results. The decomposition results include trend terms, multiple period terms, and residual terms. The trend terms and period terms are used for subsequent sequence reconstruction, and the residual terms are used for anomaly detection. The decomposition formula is: , where For the ecological environment indicators; For the Ecological environment indicators in time Observed values ​​of For the The trend items of ecological environment indicators are extracted using the LOESS smoothing method; For the The first ecological environment indicator Periodic items; For the Length of cycle; Number the cycles; is the total number of period items; For the The residual term of the ecological environment indicator.

[0007] Preferably, the specific process of step S2 is: S21. Input the residual list obtained in step S1. Each residual sequence corresponds to an ecological environment indicator. Standardize each residual sequence independently. The calculation formula is: , where is the standardized residual sequence; is the original residual sequence obtained by MSTL decomposition; is the arithmetic mean of the residual sequence; is the standard deviation of the residual series; S22, initialize the isolation forest model; S23, combining the standardized residual sequence into a feature matrix, where each column is an ecological environment indicator, and inputting it into the multivariate isolation forest model for anomaly detection; S24. The multivariate isolation forest model outputs a multidimensional anomaly score and records the outlier locations of the original data.

[0008] Preferably, the specific process of step S3 is: S31. Merge the missing value positions and outlier positions of the original data to obtain the final missing value mask; then set the value of the missing value position to NaN, so that the missing status of all indicators at the same time point remains consistent, forming multiple groups of time series with missing values, recorded as series group A; S32. Randomly select 15% of the non-missing portion of sequence group A as the validation set, and the remaining 85% of the non-missing portion of sequence group A as the training set; S33. Set the search space of MSTL hyperparameters to 0.5 to 2.0 times the default value. The hyperparameters include the periodic smoothing window. , trend item smoothing window and the polynomial order of the periodic term ; S34. Using the Bayesian optimization evaluation function, the calculation formula is: , where The objective function for Bayesian optimization; is the optimization parameter vector; is the total number of sample time points; For the validation set at time The actual observed value of Based on parameters The estimated value of the trend term; For the The estimated value of the period term; is the regularization coefficient; is the regularization term; S35. On the training set, perform hyperparameter optimization for each ecological environment indicator independently, and use the current hyperparameters to perform MSTL decomposition; S36. Multiple indicators are interpolated independently using their respective decomposition results. The trend term uses linear interpolation, and the cycle term uses the same-phase median interpolation. The calculation formula is: , ， Where, No. Ecological environment indicators in time The interpolation result of the trend term at ; and Time The adjacent known observation time points; and Time The values ​​of the adjacent known trend items; For the Ecological environment indicators in time The interpolation result of periodic terms at ; is the conditional median function; For the Ecological environment indicators at a known time point The first The value of the periodic term; is the set of all known observation time points; For time Known time points that are in phase; S37, reconstruct the sequence to obtain sequence group B, and calculate the root mean square error RMSE of each ecological environment indicator of the reconstructed sequence group B at the verification position, and output the negative value of the error. The reconstruction calculation formula is: , where For the Ecological environment indicators in time The reconstruction value of For the Interpolation results of periodic terms; S38. Use Bayesian optimization tools to find the optimal hyperparameters for each ecological environment indicator within the set search space; S39. After the Bayesian optimization is completed, the hyperparameter combination of each ecological environment indicator that maximizes the objective function value is recorded to obtain the optimal hyperparameter combination.

[0009] Preferably, the specific process of step S4 is: S41, restore the verification set position in sequence group B to the original value to obtain sequence group C; S42. Using the optimal hyperparameters of each ecological environment indicator, perform MSTL decomposition on the obtained sequence group A containing missing values; S43. Multiple indicators are interpolated independently using their respective decomposition results, where the trend term is interpolated using linear interpolation and the period term is interpolated using the same phase mean; S44. Superimpose the interpolated trend term and period term sequences to reconstruct a complete ecological environment indicator time series.

[0010] By adopting the above-mentioned technical solution, the present invention achieves the following beneficial effects: The proposed method for anomaly identification and missing value interpolation of multi-periodic ecological and environmental indicators achieves a breakthrough in the field of multi-periodic coupled time series data processing by innovatively integrating multi-periodic time series decomposition, multivariate anomaly detection, and Bayesian-driven hyperparameter optimization techniques. This method utilizes MSTL decomposition to strip away the trend and multi-periodic characteristics of the data, generating a residual sequence containing only random fluctuations and potential anomalies. This fundamentally eliminates the interference of multi-periodic coupling on anomaly identification. The multi-indicator residuals are then fed into an isolation forest model for joint analysis, enabling simultaneous multi-indicator anomaly identification and cross-indicator collaborative anomaly pattern mining. This synergistic advantage significantly overcomes the high false positive rate associated with traditional univariate thresholding methods (such as generalized extreme studentized deviation) due to their inability to handle periodic interference and cross-indicator correlation. For missing value interpolation, the present invention innovatively proposes and fully utilizes a phase-matched median interpolation strategy based on the periodic phase. This strategy leverages the inherent robustness of the median statistic and mitigates the risk of contamination of trend and periodic term reconstructions by residual outliers or skewed data distributions. The entire process also supports independent and parallel processing of multiple indicators, efficiently adapting to the needs of batch cleaning of high-frequency, multi-period water quality data. Ultimately, this method outputs complete, accurate, and reliable reconstructed time series, providing a solid data foundation for in-depth analysis of water quality monitoring data, pollution source tracing, precise early warning, and refined water environment management, with broad prospects for widespread application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 Flowchart of the present invention; Figure 2 It is a flowchart of the present invention. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0013] like Figure 1 and Figure 2 As shown, a method for identifying anomalies and missing interpolation of multi-periodic ecological environmental indicators includes the following steps: S1. Based on MSTL, time series decomposition is performed on each ecological and environmental indicator of multi-periodic ecological and environmental data; The specific process of step S1 is: S11. Input a list of multiple multi-periodic continuous ecological and environmental indicators, set the index to the timestamp type, and record the missing value locations of the original data; S12. Select the default MSTL hyperparameters and apply MSTL decomposition to each ecological environment indicator independently to obtain the decomposition results. The decomposition results include trend terms, multiple period terms, and residual terms. The trend terms and period terms are used for subsequent sequence reconstruction, and the residual terms are used for anomaly detection. The decomposition formula is: , where For the ecological environment indicators; For the Ecological environment indicators in time Observed values ​​of For the The trend items of ecological environment indicators are extracted using the LOESS smoothing method; For the The first ecological environment indicator Periodic items; For the Length of cycle; Number the cycles; is the total number of period items; For the The residual term of the ecological environment indicator; S2, normalize the residual list obtained from the decomposition into a feature matrix and input it into the multivariate isolation forest model for outlier identification; The specific process of step S2 is: S21. Input the residual list obtained in step S1. Each residual sequence corresponds to an ecological environment indicator. Standardize each residual sequence independently. The calculation formula is: , where is the standardized residual sequence; is the original residual sequence obtained by MSTL decomposition; is the arithmetic mean of the residual sequence; is the standard deviation of the residual series; S22, initialize the isolation forest model; S23, combining the standardized residual sequence into a feature matrix, where each column is an ecological environment indicator, and inputting it into the multivariate isolation forest model for anomaly detection; S24. The multivariate isolation forest model outputs a multidimensional anomaly score and records the outlier locations of the original data; S3. Merge outliers and original missing values ​​to obtain a time series with missing values. Then use the Bayesian optimization algorithm to optimize the hyperparameters of MSTL, iteratively search, and determine the optimal hyperparameter combination. The specific process of step S3 is: S31. Merge the missing value positions and outlier positions of the original data to obtain the final missing value mask; then set the value of the missing value position to NaN, so that the missing status of all indicators at the same time point remains consistent, forming multiple groups of time series with missing values, recorded as series group A; S32. Randomly select 15% of the non-missing portion of sequence group A as the validation set, and the remaining 85% of the non-missing portion of sequence group A as the training set; S33. Set the search space of MSTL hyperparameters to 0.5 to 2.0 times the default value. The hyperparameters include the periodic smoothing window. , trend item smoothing window and the polynomial order of the periodic term ; S34. Using the Bayesian optimization evaluation function, the calculation formula is: , where The objective function for Bayesian optimization; is the optimization parameter vector; is the total number of sample time points; For the validation set at time The actual observed value of Based on parameters The estimated value of the trend term; For the The estimated value of the period term; is the regularization coefficient; is the regularization term; S35. On the training set, perform hyperparameter optimization for each ecological environment indicator independently, and use the current hyperparameters to perform MSTL decomposition; S36. Multiple indicators are interpolated independently using their respective decomposition results. The trend term uses linear interpolation, and the cycle term uses the same-phase median interpolation. The calculation formula is: , ， Where, No. Ecological environment indicators in time The interpolation result of the trend term at ; and Time The adjacent known observation time points; and Time The values ​​of the adjacent known trend items; For the Ecological environment indicators in time The interpolation result of periodic terms at ; is the conditional median function; For the Ecological environment indicators at a known time point The first The value of the periodic term; is the set of all known observation time points; For time Known time points that are in phase; S37, reconstruct the sequence to obtain sequence group B, and calculate the root mean square error RMSE of each ecological environment indicator of the reconstructed sequence group B at the verification position, and output the negative value of the error. The reconstruction calculation formula is: , where For the Ecological environment indicators in time The reconstruction value of For the Interpolation results of periodic terms; S38. Use Bayesian optimization tools to find the optimal hyperparameters for each ecological environment indicator within the set search space; S39. After the Bayesian optimization is completed, the hyperparameter combination of each ecological environment indicator that maximizes the objective function value is recorded to obtain the optimal hyperparameter combination; S4. Use the optimal hyperparameters to perform MSTL decomposition, and independently interpolate multiple indicators using their respective decomposition results to reconstruct a complete ecological environment indicator time series; The specific process of step S4 is: S41, restore the verification set position in sequence group B to the original value to obtain sequence group C; S42. Using the optimal hyperparameters of each ecological environment indicator, perform MSTL decomposition on the obtained sequence group A containing missing values; S43. Multiple indicators are interpolated independently using their respective decomposition results, where the trend term is interpolated using linear interpolation and the period term is interpolated using the same phase mean; S44. Superimpose the interpolated trend term and period term sequences to reconstruct a complete ecological environment indicator time series.

[0014] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for identifying anomalies and missing interpolation of multi-periodic ecological environmental indicators, characterized in that: The following steps are involved: S1. Based on MSTL, time series decomposition is performed on each ecological and environmental indicator of multi-periodic ecological and environmental data; S2, normalize the residual list obtained from the decomposition into a feature matrix and input it into the multivariate isolation forest model for outlier identification; S3. Merge outliers and original missing values ​​to obtain a time series with missing values. Then use the Bayesian optimization algorithm to optimize the hyperparameters of MSTL, iteratively search, and determine the optimal hyperparameter combination. S4. Use the optimal hyperparameters to perform MSTL decomposition, and independently interpolate multiple indicators using their respective decomposition results to reconstruct a complete time series of ecological environmental indicators.

2. A method for identifying anomalies and missing interpolation of multi-periodic ecological environmental indicators according to claim 1, characterized in that: The specific process of step S1 is: S11. Input a list of multiple multi-periodic continuous ecological and environmental indicators, set the index to the timestamp type, and record the missing value locations of the original data; S12. Select the default MSTL hyperparameters and apply MSTL decomposition to each ecological environment indicator independently to obtain the decomposition results; The decomposition results include trend terms, multiple period terms, and residual terms. The trend terms and period terms are used for subsequent sequence reconstruction, and the residual terms are used for anomaly detection. The decomposition formula is: , where For the ecological environment indicators; For the Ecological environment indicators in time Observed values ​​of For the The trend items of ecological environment indicators are extracted using the LOESS smoothing method; For the The first ecological environment indicator Periodic items; For the Length of cycle; Number the cycles; is the total number of period items; For the The residual term of the ecological environment indicator.

3. The method for identifying anomalies and missing interpolation of multi-periodic ecological environment indicators according to claim 1, characterized in that: The specific process of step S2 is: S21. Input the residual list obtained in step S1. Each residual sequence corresponds to an ecological environment indicator. Standardize each residual sequence independently. The calculation formula is: , where is the standardized residual sequence; is the original residual sequence obtained by MSTL decomposition; is the arithmetic mean of the residual sequence; is the standard deviation of the residual series; S22, initialize the isolation forest model; S23, combining the standardized residual sequence into a feature matrix, where each column is an ecological environment indicator, and inputting it into the multivariate isolation forest model for anomaly detection; S24. The multivariate isolation forest model outputs a multidimensional anomaly score and records the outlier locations of the original data.

4. A method for identifying anomalies and missing interpolation of multi-periodic ecological environmental indicators according to claim 1, characterized in that: The specific process of step S3 is: S31. Merge the missing value positions and outlier positions of the original data to obtain the final missing value mask; then set the value of the missing value position to NaN, so that the missing status of all indicators at the same time point remains consistent, forming multiple groups of time series with missing values, recorded as series group A; S32. Randomly select 15% of the non-missing portion of sequence group A as the validation set, and the remaining 85% of the non-missing portion of sequence group A as the training set; S33. Set the search space of MSTL hyperparameters to 0.5 to 2.0 times the default value. The hyperparameters include the periodic smoothing window. , trend item smoothing window and the polynomial order of the periodic term ; S34. Using the Bayesian optimization evaluation function, the calculation formula is: , where The objective function for Bayesian optimization; is the optimization parameter vector; is the total number of sample time points; For the validation set at time The actual observed value of Based on parameters The estimated value of the trend term; For the The estimated value of the period term; is the regularization coefficient; is the regularization term; S35. On the training set, perform hyperparameter optimization for each ecological environment indicator independently, and use the current hyperparameters to perform MSTL decomposition; S36. Multiple indicators are interpolated independently using their respective decomposition results. The trend term uses linear interpolation, and the cycle term uses the same-phase median interpolation. The calculation formula is: , ， Where, No. Ecological environment indicators in time The interpolation result of the trend term at ; and Time The adjacent known observation time points; and Time The values ​​of the adjacent known trend items; For the Ecological environment indicators in time The interpolation result of periodic terms at ; is the conditional median function; For the Ecological environment indicators at a known time point The first The value of the periodic term; is the set of all known observation time points; For time Known time points that are in phase; S37, reconstruct the sequence to obtain sequence group B, and calculate the root mean square error RMSE of each ecological environment indicator of the reconstructed sequence group B at the verification position, and output the negative value of the error. The reconstruction calculation formula is: , where For the Ecological environment indicators in time The reconstruction value of For the Interpolation results of periodic terms; S38. Use Bayesian optimization tools to find the optimal hyperparameters for each ecological environment indicator within the set search space; S39. After the Bayesian optimization is completed, the hyperparameter combination of each ecological environment indicator that maximizes the objective function value is recorded to obtain the optimal hyperparameter combination.

5. The method for identifying anomalies and missing interpolation of multi-periodic ecological environment indicators according to claim 1, characterized in that: The specific process of step S4 is: S41, restore the verification set position in sequence group B to the original value to obtain sequence group C; S42. Using the optimal hyperparameters of each ecological environment indicator, perform MSTL decomposition on the obtained sequence group A containing missing values; S43. Multiple indicators are interpolated independently using their respective decomposition results, where the trend term is interpolated using linear interpolation and the period term is interpolated using the same phase mean; S44. Superimpose the interpolated trend term and period term sequences to reconstruct a complete ecological environment indicator time series.

Images