A multi-periodicity ecological environment index anomaly identification and missing interpolation method

Abstract

The application discloses a multi-periodicity ecological environment index anomaly identification and missing value interpolation method, comprising the following steps: S1, based on MSTL, time series decomposition is respectively performed on each ecological environment index of multi-periodicity ecological environment data; S2, the residual list obtained by decomposition is standardized into a feature matrix, and is input into a multivariate isolation forest model for anomaly value identification; S3, the anomaly value and the original missing value are combined to obtain a time series containing missing values, and a Bayesian optimization algorithm is used to optimize the hyperparameters of MSTL, and the optimal hyperparameter combination is determined through iterative search; S4, MSTL decomposition is performed by using the optimal hyperparameters, multi-index independent interpolation is used, and a complete ecological environment index time series is reconstructed; the method supports multi-index parallel processing, can accurately identify anomaly values in water quality data, and can perform high-precision interpolation by using the periodic characteristics of data, effectively improves the quality of water quality data, and has a good popularization and application prospect.

Description

Technical Field

[0001] This invention belongs to the field of water quality monitoring and management technology, specifically relating to a method for identifying and imputing missing multi-periodic ecological and environmental indicators. Background Technology

[0002] Currently, high-frequency, multi-period automatic water quality monitoring technology, as a key means of water environment management, has been widely applied in key river basins. The widespread establishment of automatic water quality monitoring stations provides a high-frequency, multi-period data foundation for identifying the dynamic patterns of water quality parameters. Their dense monitoring capabilities, ranging from minutes to hours, can accurately capture the spatiotemporal evolution characteristics of physicochemical indicators such as dissolved oxygen and water temperature, becoming a crucial support for precise pollution control in river basins. However, continuous instrument operation is susceptible to environmental interference, leading to widespread omissions, errors, and outliers in the data. Traditional manual verification methods are difficult to implement due to the massive amount of data.

[0003] The unique characteristics of multi-period data exacerbate the difficulty of data cleaning: on the one hand, automated monitoring instruments are susceptible to environmental interference during continuous operation, and missing segments often exhibit non-random distributions. More importantly, this type of data possesses strong autocorrelation and multi-period coupling characteristics, making conventional anomaly detection methods such as the generalized extreme student biochemical bias method prone to misjudging natural fluctuations as anomalies. On the other hand, the complex nonlinear relationships between variables lead to traditional interpolation methods generating results that violate physical laws. While existing research attempts to introduce predictive models for real-time anomaly screening, it focuses on short-term forecasting rather than full-sequence cleaning and fails to address the issues of period decoupling and embedding physical mechanisms, making it difficult to meet the requirements for data authenticity, accuracy, and comprehensiveness. Therefore, there is an urgent need to develop batch cleaning methods for indicator anomaly identification and missing data imputation that are adapted to multi-period characteristics and integrate temporal decomposition and mechanistic constraints. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a method for identifying and imputing missing values ​​in multi-periodic ecological and environmental indicators. This method supports parallel processing of multiple indicators, accurately identifies outliers in water quality data, and utilizes the periodicity of the data for high-precision interpolation, effectively improving the quality of water quality data and demonstrating promising prospects for widespread application.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for identifying and imputing missing ecological and environmental indicators across multiple periods includes the following steps:

[0007] S1. Based on MSTL, perform time series decomposition on each ecological and environmental indicator of multi-periodic ecological and environmental data;

[0008] S2. Standardize the list of residuals obtained from the decomposition into a feature matrix and input it into the multivariate isolated forest model for outlier identification;

[0009] S3. Merge outliers and original missing values ​​to obtain a time series containing missing values. Then, use the Bayesian optimization algorithm to optimize the hyperparameters of MSTL, iteratively search and determine the optimal combination of hyperparameters.

[0010] S4. Use the optimal hyperparameters to perform MSTL decomposition, and use the decomposition results of each index to independently interpolate and reconstruct the complete time series of ecological and environmental indicators.

[0011] Preferably, the specific process of step S1 is as follows:

[0012] S11. Input a list of multiple multi-periodic continuous ecological and environmental indicators, set the index to timestamp type, and record the location of missing values ​​in the original data;

[0013] S12. Select the default MSTL hyperparameters and apply MSTL decomposition independently to each ecological and environmental indicator to obtain the decomposition results. The decomposition results include a trend term, multiple periodic terms, and residual terms. Among them, the trend term and periodic terms are used for subsequent sequence reconstruction, and the residual terms are used for anomaly detection. The decomposition formula is: In the formula, For the first One ecological and environmental indicator; For the first Each ecological and environmental indicator over time Observed values; For the first The trend terms of each ecological and environmental indicator were extracted using the LOESS smoothing method. For the first The first ecological and environmental indicator One periodic item; For the first The length of each cycle; For periodic numbering; This represents the total number of periodic terms. For the first The residual terms of each ecological and environmental indicator.

[0014] Preferably, the specific process of step S2 is as follows:

[0015] S21. Input the residual list obtained in step S1. Each residual sequence corresponds to an ecological environment indicator. Standardize each residual sequence independently. The calculation formula is: In the formula, The standardized residual sequence; The original residual sequence obtained from MSTL decomposition; This represents the arithmetic mean of the residual sequence; The standard deviation of the residual sequence;

[0016] S22. Initialize the isolated forest model;

[0017] S23. Combine the standardized residual sequences into a feature matrix, where each column represents an ecological environment indicator, and input it into a multivariate isolated forest model for anomaly detection.

[0018] S24. The multivariate isolated forest model outputs multidimensional outlier scores and records the outlier locations in the original data.

[0019] Preferably, the specific process of step S3 is as follows:

[0020] S31. Merge the missing value locations and outlier locations of the original data to obtain the final missing value mask; then set the missing value location to NaN, so that the missing status of all indicators is consistent at the same time point, forming multiple time series containing missing values, denoted as sequence group A;

[0021] S32. Randomly select 15% of the non-missing parts of sequence group A as the validation set, and the remaining 85% of the non-missing parts of sequence group A as the training set.

[0022] S33. Set the search space for MSTL hyperparameters to 0.5 to 2.0 times the default value. Hyperparameters include the periodic term smoothing window. Trend Item Smoothing Window and the order of periodic polynomials ;

[0023] S34. The Bayesian optimization evaluation function is used, and the calculation formula is as follows: In the formula, Optimize the objective function using Bayesian methods; To optimize the parameter vector; This represents the total number of sample time points; To verify the set in time The actual observed values; For parameter-based The estimated value of the trend term; For the first Estimates of the periodic terms; The regularization coefficient is used. For regularization terms;

[0024] S35. On the training set, perform hyperparameter optimization independently for each ecological environment indicator and use the current hyperparameters for MSTL decomposition.

[0025] S36. Multiple indicators are interpolated independently using their respective decomposition results. The trend term uses linear interpolation, and the period term uses in-phase median interpolation. The calculation formula is as follows: , ， In the formula, No. Each ecological and environmental indicator over time interpolation results for the trend term at the location; and They are time Known observation time points that are adjacent to each other;

[0026] and They are time The values ​​of adjacent known trend terms; For the first Each ecological and environmental indicator over time Interpolation results of the periodic terms at the specified location; This is a function of the conditional median; For the first The ecological and environmental indicators at a known time point The first The value of each periodic term; It is the set of all known observation time points; In order to keep pace with time Known time points in phase;

[0027] S37. Reconstruct the sequence to obtain sequence group B, and calculate the root mean square error (RMSE) of each ecological and environmental indicator of the reconstructed sequence group B at the verification location. Output the negative value of the error. The reconstruction calculation formula is as follows: In the formula, For the first Each ecological and environmental indicator over time The reconstructed value; For the first Interpolation results for each periodic term;

[0028] S38. Use Bayesian optimization tools to find the optimal hyperparameters for each ecological environment indicator within the set search space;

[0029] S39. After Bayesian optimization, record the hyperparameter combination that maximizes the objective function value for each ecological and environmental indicator to obtain the optimal hyperparameter combination.

[0030] Preferably, the specific process of step S4 is as follows:

[0031] S41. Restore the validation set positions in sequence group B to their original values ​​to obtain sequence group C;

[0032] S42. Using the optimal hyperparameters of each ecological and environmental indicator, perform MSTL decomposition on the obtained sequence group A containing missing values;

[0033] S43. Multiple indicators are interpolated independently using their respective decomposition results. The trend term is interpolated linearly, and the period term is interpolated using the mean of the same phase.

[0034] S44. Overlay the interpolated trend and periodic sequences to reconstruct a complete time series of ecological and environmental indicators.

[0035] After adopting the above technical solution, the present invention has the following beneficial effects: The multi-periodic ecological environment indicator anomaly identification and missing value imputation method provided by the present invention has achieved a breakthrough in the field of multi-periodic coupled time series data processing by innovatively integrating multi-periodic time series decomposition, multivariate anomaly detection and Bayesian-driven hyperparameter optimization technology. This method uses MSTL decomposition to remove the trend and multi-periodic features of the data, generating a residual sequence containing only random fluctuations and potential anomalies, fundamentally eliminating the interference of multi-periodic coupling on anomaly discrimination. Then, the multi-indicator residuals are input into an isolated forest model for joint analysis, realizing simultaneous identification of multi-indicator anomalies and cross-indicator collaborative anomaly pattern mining. The two work together to form a cascade advantage, significantly overcoming the high misjudgment rate defect of traditional univariate threshold methods (such as generalized extreme studentization bias) due to their inability to handle periodic interference and cross-indicator correlation. In the missing value imputation stage, the present invention creatively proposes and adopts the in-phase median interpolation strategy based on periodic phase throughout the process, making full use of the natural robustness of the median statistic, avoiding the pollution risk of residual outliers or skewed data distribution on the reconstruction of trend and periodic terms. Meanwhile, the entire process supports parallel and independent processing of multiple indicators, efficiently adapting to the batch cleaning needs of high-frequency, multi-period water quality data. Ultimately, this method outputs a complete, accurate, and reliable reconstructed time series, providing a solid data foundation for in-depth analysis of water quality monitoring data, pollution source tracing, precise early warning, and refined water environment management, and has broad prospects for widespread application. Attached Figure Description

[0036] Figure 1 This is a flowchart of the present invention;

[0037] Figure 2 This is a flowchart of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0039] like Figure 1 and Figure 2 As shown, a method for identifying and imputing missing multi-periodic ecological and environmental indicators includes the following steps:

[0040] S1. Based on MSTL, perform time series decomposition on each ecological and environmental indicator of multi-periodic ecological and environmental data;

[0041] The specific process of step S1 is as follows:

[0042] S11. Input a list of multiple multi-periodic continuous ecological and environmental indicators, set the index to timestamp type, and record the location of missing values ​​in the original data;

[0043] S12. Select the default MSTL hyperparameters and apply MSTL decomposition independently to each ecological and environmental indicator to obtain the decomposition results. The decomposition results include a trend term, multiple periodic terms, and residual terms. Among them, the trend term and periodic terms are used for subsequent sequence reconstruction, and the residual terms are used for anomaly detection. The decomposition formula is: In the formula, For the first One ecological and environmental indicator; For the first Each ecological and environmental indicator over time Observed values; For the first The trend terms of each ecological and environmental indicator were extracted using the LOESS smoothing method. For the first The first ecological and environmental indicator One periodic item; For the first The length of each cycle; For periodic numbering; This represents the total number of periodic terms. For the first The residual terms of each ecological and environmental indicator;

[0044] S2. Standardize the list of residuals obtained from the decomposition into a feature matrix and input it into the multivariate isolated forest model for outlier identification;

[0045] The specific process of step S2 is as follows:

[0046] S21. Input the residual list obtained in step S1. Each residual sequence corresponds to an ecological environment indicator. Standardize each residual sequence independently. The calculation formula is: In the formula, The standardized residual sequence; The original residual sequence obtained from MSTL decomposition; This represents the arithmetic mean of the residual sequence; The standard deviation of the residual sequence;

[0047] S22. Initialize the isolated forest model;

[0048] S23. Combine the standardized residual sequences into a feature matrix, where each column represents an ecological environment indicator, and input it into a multivariate isolated forest model for anomaly detection.

[0049] S24. The multivariate isolated forest model outputs multidimensional outlier scores and records the outlier locations in the original data.

[0050] S3. Merge outliers and original missing values ​​to obtain a time series containing missing values. Then, use the Bayesian optimization algorithm to optimize the hyperparameters of MSTL, iteratively search and determine the optimal combination of hyperparameters.

[0051] The specific process of step S3 is as follows:

[0052] S31. Merge the missing value locations and outlier locations of the original data to obtain the final missing value mask; then set the missing value location to NaN, so that the missing status of all indicators is consistent at the same time point, forming multiple time series containing missing values, denoted as sequence group A;

[0053] S32. Randomly select 15% of the non-missing parts of sequence group A as the validation set, and the remaining 85% of the non-missing parts of sequence group A as the training set.

[0054] S33. Set the search space for MSTL hyperparameters to 0.5 to 2.0 times the default value. Hyperparameters include the periodic term smoothing window. Trend Item Smoothing Window and the order of periodic polynomials ;

[0055] S34. The Bayesian optimization evaluation function is used, and the calculation formula is as follows: In the formula, Optimize the objective function using Bayesian methods; To optimize the parameter vector; This represents the total number of sample time points; To verify the set in time The actual observed values; For parameter-based The estimated value of the trend term; For the first Estimates of the periodic terms; The regularization coefficient is used. For regularization terms;

[0056] S35. On the training set, perform hyperparameter optimization independently for each ecological environment indicator and use the current hyperparameters for MSTL decomposition.

[0057] S36. Multiple indicators are interpolated independently using their respective decomposition results. The trend term uses linear interpolation, and the period term uses in-phase median interpolation. The calculation formula is as follows: , ， In the formula, No. Each ecological and environmental indicator over time interpolation results for the trend term at the location; and They are time Known observation time points that are adjacent to each other;

[0058] and They are time The values ​​of adjacent known trend terms; For the first Each ecological and environmental indicator over time Interpolation results of the periodic terms at the specified location; This is a function of the conditional median; For the first The ecological and environmental indicators at a known time point The first The value of each periodic term; It is the set of all known observation time points; In order to keep pace with time Known time points in phase;

[0059] S37. Reconstruct the sequence to obtain sequence group B, and calculate the root mean square error (RMSE) of each ecological and environmental indicator of the reconstructed sequence group B at the verification location. Output the negative value of the error. The reconstruction calculation formula is as follows: In the formula, For the first Each ecological and environmental indicator over time The reconstructed value; For the first Interpolation results for each periodic term;

[0060] S38. Use Bayesian optimization tools to find the optimal hyperparameters for each ecological environment indicator within the set search space;

[0061] S39. After Bayesian optimization, record the hyperparameter combination that maximizes the objective function value for each ecological and environmental indicator to obtain the optimal hyperparameter combination.

[0062] S4. Use the optimal hyperparameters to perform MSTL decomposition, and interpolate the multiple indicators independently using their respective decomposition results to reconstruct a complete time series of ecological and environmental indicators.

[0063] The specific process of step S4 is as follows:

[0064] S41. Restore the validation set positions in sequence group B to their original values ​​to obtain sequence group C;

[0065] S42. Using the optimal hyperparameters of each ecological and environmental indicator, perform MSTL decomposition on the obtained sequence group A containing missing values;

[0066] S43. Multiple indicators are interpolated independently using their respective decomposition results. The trend term is interpolated linearly, and the period term is interpolated using the mean of the same phase.

[0067] S44. Overlay the interpolated trend and periodic sequences to reconstruct a complete time series of ecological and environmental indicators.

[0068] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for identifying and imputing missing multi-periodic ecological and environmental indicators, characterized in that, Includes the following steps: S1. Based on MSTL, perform time series decomposition on each ecological and environmental indicator of multi-periodic ecological and environmental data; S2. Standardize the list of residuals obtained from the decomposition into a feature matrix and input it into the multivariate isolated forest model for outlier identification; The specific process of step S2 is as follows: S21. Input the residual list obtained in step S1. Each residual sequence corresponds to an ecological environment indicator. Standardize each residual sequence independently. The calculation formula is: In the formula, The standardized residual sequence; The original residual sequence obtained from MSTL decomposition; This represents the arithmetic mean of the residual sequence; The standard deviation of the residual sequence; S22. Initialize the isolated forest model; S23. Combine the standardized residual sequences into a feature matrix, where each column represents an ecological environment indicator, and input it into a multivariate isolated forest model for anomaly detection. S24. The multivariate isolated forest model outputs multidimensional outlier scores and records the outlier locations in the original data. S3. Merge outliers and original missing values ​​to obtain a time series containing missing values. Then, use the Bayesian optimization algorithm to optimize the hyperparameters of MSTL, iteratively search and determine the optimal combination of hyperparameters. The specific process of step S3 is as follows: S31. Merge the missing value locations and outlier locations of the original data to obtain the final missing value mask; Then set the missing value position to NaN. The missing status of all indicators is consistent at the same time point, forming multiple time series with missing values, which are denoted as sequence group A. S32. Randomly select 15% of the non-missing parts of sequence group A as the validation set, and the remaining 85% of the non-missing parts of sequence group A as the training set. S33. Set the search space for MSTL hyperparameters to 0.5 to 2.0 times the default value. Hyperparameters include the periodic term smoothing window. Trend Item Smoothing Window and the order of periodic polynomials ; S34. Use Bayesian optimization of the evaluation function; S35. On the training set, perform hyperparameter optimization independently for each ecological environment indicator and use the current hyperparameters for MSTL decomposition. S36. Multiple indicators are interpolated independently using their respective decomposition results. The trend term is interpolated linearly, and the period term is interpolated using the median of the same phase. S37. Reconstruct the sequence to obtain sequence group B, and calculate the root mean square error (RMSE) of each ecological and environmental indicator of the reconstructed sequence group B at the verification location, and output the negative value of the error. S38. Use Bayesian optimization tools to find the optimal hyperparameters for each ecological environment indicator within the set search space; S39. After Bayesian optimization, record the hyperparameter combination that maximizes the objective function value for each ecological and environmental indicator to obtain the optimal hyperparameter combination. S4. Use the optimal hyperparameters to perform MSTL decomposition, and use the decomposition results of each index to independently interpolate and reconstruct the complete time series of ecological and environmental indicators.

2. The method for identifying and imputing missing multi-periodic ecological and environmental indicators as described in claim 1, characterized in that, The specific process of step S1 is as follows: S11. Input a list of multiple multi-periodic continuous ecological and environmental indicators, set the index to timestamp type, and record the location of missing values ​​in the original data; S12. Select the default MSTL hyperparameters and apply MSTL decomposition independently to each ecological and environmental indicator to obtain the decomposition results; The decomposition result includes a trend term, multiple periodic terms, and a residual term; among them, the trend term and periodic term are used for subsequent sequence reconstruction, and the residual term is used for anomaly detection. The decomposition formula is: In the formula, For the first One ecological and environmental indicator; For the first Each ecological and environmental indicator over time Observed values; For the first The trend terms of each ecological and environmental indicator were extracted using the LOESS smoothing method. For the first The first ecological and environmental indicator One periodic item; For the first The length of each cycle; For periodic numbering; This represents the total number of periodic terms. For the first The residual terms of each ecological and environmental indicator.

3. The method for identifying and imputing missing multi-periodic ecological and environmental indicators as described in claim 1, characterized in that: In step S34, the Bayesian optimization evaluation function is calculated using the following formula: In the formula, Optimize the objective function using Bayesian methods; To optimize the parameter vector; This represents the total number of sample time points; To verify the set in time The actual observed values; For parameter-based The estimated value of the trend term; For the first Estimates of the periodic terms; The regularization coefficient is used. For regularization terms; In step S36, the calculation formula for independent interpolation of multiple indicators using their respective decomposition results is as follows: , ， In the formula, No. Each ecological and environmental indicator over time interpolation results for the trend term at the location; and They are time Known observation time points that are adjacent to each other; and They are time The values ​​of adjacent known trend terms; For the first Each ecological and environmental indicator over time Interpolation results of the periodic terms at the specified location; This is a function of the conditional median; For the first The ecological and environmental indicators at a known time point The first The value of each periodic term; It is the set of all known observation time points; In order to keep pace with time Known time points in phase; In step S37, the reconstruction calculation formula is: In the formula, For the first Each ecological and environmental indicator over time The reconstructed value; For the first Interpolation results for each periodic term.

4. The method for identifying and imputing missing multi-periodic ecological and environmental indicators as described in claim 1, characterized in that, The specific process of step S4 is as follows: S41. Restore the validation set positions in sequence group B to their original values ​​to obtain sequence group C; S42. Using the optimal hyperparameters of each ecological and environmental indicator, perform MSTL decomposition on the obtained sequence group A containing missing values; S43. Multiple indicators are interpolated independently using their respective decomposition results. The trend term is interpolated linearly, and the period term is interpolated using the mean of the same phase. S44. Overlay the interpolated trend and periodic sequences to reconstruct a complete time series of ecological and environmental indicators.

Images