Water level time sequence data cleaning method based on regression statistics

Abstract

The invention discloses a water level time sequence data cleaning method based on regression statistics, which comprises the following steps: S1, sorting flood data of a historical session of a monitoring station to form a complete flood sample set; s2, establishing a function model according to the nonlinear change characteristics of the flood water level; s3, performing regression function fitting on the historical flood sample data, and calculating linear correlation between each fitting function and actual data; s4, based on a fitting result of the optimal regression function, establishing a Gaussian distribution model of a difference value; s5, determining a confidence interval of Gaussian distribution through the verification sample, and judging whether the data is abnormal data or not according to the confidence interval; s6, the detected abnormal data are corrected, and the cleaned data are obtained; and S7, processing the first N time sequence points in the flood time sequence data by adopting a moving average algorithm, and eliminating the interference of accidental fluctuation on regression fitting. The method has the advantages of being high in cleaning precision, high in adaptability and good in flood nonlinear change characteristic processing effect.

Description

Technical Field

[0001] The present invention relates to the technical field of water level time series data cleaning, and in particular to a water level time series data cleaning method based on regression statistics. Background Art

[0002] With the rapid development of information technology and intelligent sensor technology, flood forecasting has become an important research direction in the field of water conservancy. However, due to the sensor manufacturing process, complex environmental factors and interference in the data transmission process, the water level time series data collected by the sensor often contains abnormal data points, which may seriously affect the accuracy and reliability of the flood forecasting model. As a key expression of the flood process, the accuracy of flood water level data is directly related to the effectiveness of disaster prevention and mitigation work.

[0003] In the prior art, the cleaning methods for water level time series data mainly include smoothing-based, constraint-based and statistics-based cleaning techniques. These methods have the following obvious deficiencies when processing abnormal points in flood water level data:

[0004] 1. Smoothing methods are prone to data distortion: Although traditional smoothing methods can quickly eliminate data fluctuations, they are prone to excessive modification of the original data, resulting in distortion and loss of correct data, ultimately affecting the accuracy of flood time series data.

[0005] 2. Constraint-based methods rely on domain knowledge: Constraint-based methods usually require people to set specific rules and parameters to limit and filter data. However, the characteristics of different flood processes and river basins are complex and changeable, and a single constraint rule is difficult to universalize. In addition, without the support of domain experts, this method is difficult to implement effectively.

[0006] 3. Statistical methods are difficult to dynamically adapt to nonlinear changes: Traditional statistical methods usually use a single regression model and fixed statistical rules, which are difficult to cope with the nonlinear characteristics and complex changes of flood water levels. Especially at different stages of flood development, a single model often cannot accurately capture the law of water level changes.

[0007] 4. Lack of intelligent abnormal data correction mechanism: Existing technologies usually simply eliminate and replace abnormal data points, but fail to make full use of historical data and regression models to accurately correct abnormal data, which affects the continuity and integrity of the data.

[0008] Therefore, how to provide a water level time series data cleaning method based on regression statistics is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0009] One purpose of the present invention is to propose a water level time series data detection and cleaning method based on regression statistics. The present invention makes full use of regression statistical analysis, Gaussian distribution modeling, sliding average algorithm and nonlinear regression model, and describes in detail the technical solution for cleaning water level time series data by constructing regression function clusters, multi-stage dynamic regression fitting and abnormal data correction. The method has the advantages of high cleaning accuracy, strong adaptability and good processing effect on nonlinear change characteristics of floods.

[0010] A method for detecting and cleaning water level time series data based on regression statistics according to an embodiment of the present invention is characterized by comprising the following steps:

[0011] S1. Sort out the historical flood data of the monitoring station, extract the water level time series data of each flood, and perform sliding time series processing on it to form a complete flood sample set;

[0012] S2. According to the nonlinear variation characteristics of flood water level, a regression function cluster is established, including quadratic curve, exponential curve, modified exponential curve, CompertZ curve and Logistic curve;

[0013] S3, fitting the historical flood sample data with a regression function, using the curve linearization method to calculate the linear correlation between each fitting function and the actual data, and the regression function with the highest correlation is taken as the optimal regression function;

[0014] S4. Based on the fitting result of the optimal regression function, the difference between the predicted value and the measured value is calculated, and a statistical analysis is performed on the difference to establish a Gaussian distribution model of the difference;

[0015] S5. Determine the confidence interval of the Gaussian distribution through the verification sample, and determine whether the data is abnormal data based on the confidence interval;

[0016] S6. For the detected abnormal data, according to the regression function parameters and the Euclidean distance calculation, the regression function with the smallest historical fitting parameter distance is selected to correct the abnormal data to obtain the cleaned data;

[0017] S7. The first N time series points in the flood time series data are processed using a sliding average algorithm to eliminate the interference of accidental fluctuations on regression fitting.

[0018] Optionally, the S1 specifically includes:

[0019] S11. Set the flood limit water level Z of the monitoring station f , traverse the historical water level time series data of the monitoring station, and start recording the data of this period when the water level sequence shows a monotonically increasing trend;

[0020] S12. Detect temporary peak value Z in water level time series data tmp, when Z tmp >Z f When a flood occurs, it is determined that there is a flood process, and the flood starting point and flood peak point are determined by tracing back from the temporary peak.

[0021] S13, starting from the flood peak point, traverse the water level time series data one by one backward until the water level drops and loses monotonicity, mark the flood end point, and thus extract the complete flood water level time series process;

[0022] S14. For the extracted flood water level time series data, the water level data is processed using a sliding average algorithm to calculate the sliding average of each time series point. The sliding window size is N:

[0023]

[0024] in, is the sliding average of the i-th time point, Z j is the original water level value at the jth time point, N is the sliding window size, and the appropriate N value is set according to the historical flood water level change law;

[0025] S15. After the sliding average processing, a continuous and smooth time-series water level sample set is formed for each flood process.

[0026] Optionally, the S2 specifically includes:

[0027] S21. Analyze the nonlinear variation of flood water levels, determine the characteristics of flood water levels in the rising and falling stages, including the acceleration and deceleration trends of water level changes, and form a function cluster demand analysis;

[0028] S22, establish a cluster of regression functions, including quadratic curve, exponential curve, modified exponential curve, CompertZ curve and Logistic curve;

[0029] S23. When establishing a regression function cluster, the fitting effect of historical flood data is analyzed, the function types that are not suitable for describing the flood process are eliminated, and the composition of the function cluster is dynamically adjusted so that the composition of the regression function cluster can cover the diversity of flood water level changes.

[0030] Optionally, the S3 specifically includes:

[0031] S31, select the water level data of the first N time series points from the historical flood sample data to form a time series data sequence {Z1, Z2, ..., Z N};

[0032] S32, for each regression function in the regression function cluster, use the linearization method to transform the time series data sequence {Z1, Z2, …, Z N} to convert and obtain the corresponding linearized data sequence;

[0033] S33. Calculate the linear correlation between the original time series data sequence and each regression function linearized data sequence:

[0034]

[0035] Among them, R xy is the linear correlation coefficient, x i and i are the values ​​of the time series data points before and after linearization, and x i and i The mean of

[0036] S34, comparing the linear correlation coefficients of different regression functions, selecting the regression function corresponding to the maximum absolute value of the correlation coefficient as the optimal regression function of the current time series point, repeating the above calculation process point by point for the fitting process of the entire time series data, recording the optimal regression function corresponding to each time series point, and generating an optimal regression function sequence;

[0037] S35. Based on the generated optimal regression function sequence, the prediction value of each time series point is calculated by combining the original time series data and the fitting result of the optimal regression function sequence.

[0038]

[0039] in, is the predicted value of the i-th time point, f(x i ; θ) is the optimal regression function corresponding to the i-th time point, and θ is the parameter of the optimal regression function.

[0040] Optionally, the S4 specifically includes:

[0041] S41. Prediction value for each time point in historical flood sample data and the measured value Z i Perform difference calculation:

[0042]

[0043] Among them, ΔZ i is the difference of the i-th time point, is the predicted value calculated according to the optimal regression function, Z i is the actual measured value;

[0044] S42, for all the differences {ΔZ1, ΔZ2, ..., ΔZ N}, use statistical methods to calculate its mean μ and variance σ of the difference 2 :

[0045]

[0046] Among them, N is the total number of difference sample points, μ is the mean of the difference, σ 2 is the variance of the difference;

[0047] S43. Based on the mean and variance of the difference, a Gaussian distribution model is constructed:

[0048]

[0049] Among them, p(x) is the probability density of Gaussian distribution at point x, μ is the mean of the difference, σ 2 is the variance of the difference;

[0050] S44. According to the Gaussian distribution model, the range of the difference distribution is calculated by verifying the sample and the confidence interval [-v,v] is determined:

[0051] v = μ ± kσ;

[0052] Wherein, v is the upper limit of the difference range that meets a certain confidence level, and k is the confidence factor, which is determined according to the set confidence level. The value of k=1.96 corresponds to a confidence level of 95%.

[0053] Optionally, the S5 specifically includes:

[0054] S51. Based on the Gaussian distribution model and the confidence interval of the difference [-v, v], each time series point in the flood time series data is tested to determine the predicted value of the current time series point. and the measured value Z i The difference ΔZ i Whether it exceeds the confidence interval range;

[0055] S52, when ΔZ i Satisfy v≤ΔZ i ≤v, the data at the current timing point is considered normal. i If the above conditions are not met, that is, ΔZi<-v and ΔZi>v, the current time sequence point is determined to be abnormal data, and the time sequence number of the abnormal point and its corresponding difference Z are recorded. i ;

[0056] S53: For the detected abnormal data, mark the current time series point data as abnormal, retain the upper and lower adjacent data points of the flood time series, and record the measurement value Z of the abnormal data point. i and the corresponding predicted value

[0057]

[0058] S54, for each abnormal data point, the difference ΔZ i Perform classification statistics to generate abnormal data distribution characteristics, including the mean and variance of abnormal point difference values ​​and the distribution of offsets outside the confidence interval;

[0059] S56, separating the detected abnormal data from the normal data to form two independent data sets, namely the normal data set Z normal and the abnormal data set Z anomaly .

[0060] Optionally, the S6 specifically includes:

[0061] S61, for the detected abnormal data point Z anomaly , extract the corresponding optimal regression function parameter set θ i , where θ i =(θ i1 ,θ i2 ,,θ in ) represents the regression function parameter of the i-th abnormal data point;

[0062] S62. Extract the historical parameter set of the same function type from the historical regression fitting results

[0063] {θ h1 ,θ h2 ,...,θ hm}, where m is the number of functions of the same type in the history matching results;

[0064] S63, calculate the regression parameter θ of the abnormal data point i With the historical parameter set θ hj The Euclidean distance d j :

[0065]

[0066] Among them, θ ik and θ hjk are the values ​​of the kth parameter in the abnormal data point and the historical fitting result, respectively, and d j is the Euclidean distance between the outlier parameter and the jth historical parameter;

[0067] S64. Among all historical parameters, find the parameter θ with the smallest Euclidean distance hmin , and its corresponding historical fitting prediction value is f(x i θ hmin ), where f(x i ; θ) represents the predicted value of the optimal regression function;

[0068] S65, predicting the value based on the current regression function of the abnormal data point The predicted value f(x i θ hmin ), correct the measured values ​​of abnormal data points:

[0069] Z cleaned =f(x i θ hmin );

[0070] Among them, Z cleaned is the cleaned value corrected for abnormal data points;

[0071] S66, the cleaned data Z cleaned Replace the original abnormal data Z anomaly , while retaining the regression function parameters and calculation details in the correction process, verification and model optimization, repeating the correction process for all detected abnormal data points until the entire flood time series data is cleaned and the final cleaned time series data set Z is generated final .

[0072] Optionally, the S7 specifically includes:

[0073] S71. For flood time series data {Z1, Z2, ..., Z N} perform sliding average processing on the first M time points in the sequence, and calculate the sliding average value of each time point

[0074] S72, setting a sliding window size k, and determining a value range of k according to the fluctuation characteristics of historical flood water level data;

[0075] S73, for the data points {Z1, Z2, ..., Z k}, complete the window by repeatedly expanding the existing data points and calculate the sliding average of the initial time series points:

[0076]

[0077] Where i is the number of the current time series point, and j is the number of the water level data point in the starting window;

[0078] S74. After the sliding average processing is completed, a smoothed water level data sequence is generated. During the sliding average processing, the processed data sequence is detected in real time Is there a mutation? If abnormal fluctuations are detected, record the abnormal points and keep the original data Z i .

[0079] The beneficial effects of the present invention are:

[0080] (1) The present invention proposes a dynamically adaptive water level time series data cleaning method by combining regression statistical analysis, Gaussian distribution modeling, sliding average algorithm and nonlinear regression model, which effectively copes with the characteristics of nonlinear changes and complex fluctuations of flood water levels, can accurately identify and correct abnormal data points, and avoids the error accumulation and data distortion problems caused by simple elimination and over-smoothing of abnormal data in traditional methods.

[0081] (2) The present invention adopts regression function cluster and dynamic regression fitting technology, selects the optimal regression function according to the changing law of flood water level in different development stages, improves the fitting accuracy of flood rising, peak and falling stages, and significantly improves the flexibility and accuracy of data cleaning.

[0082] (3) The present invention constructs a difference Gaussian distribution model and dynamically adjusts the confidence interval, so that the intensity of data cleaning can be flexibly adjusted according to actual needs, which not only ensures the accuracy of data cleaning but also adapts to the business needs of different scenarios.

[0083] (4) The present invention introduces a sliding average algorithm to eliminate the interference of accidental fluctuations on regression fitting and anomaly detection. At the same time, it combines Euclidean distance calculation to accurately correct abnormal data, ensuring the continuity and integrity of time series data after data cleaning.

[0084] (5) While realizing full-time anomaly detection and cleaning, the present invention provides a high-quality data basis for the optimization and verification of subsequent flood forecasting models by storing historical data and correction parameters, thereby greatly improving the accuracy and reliability of flood forecasting. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0086] Figure 1 A statistical model diagram for establishing abnormal data judgment for a method for detecting and cleaning water level time series data based on regression statistics proposed by the present invention;

[0087] Figure 2 This is a diagram of outlier detection and cleaning of flood data according to a method for detecting and cleaning water level time series data based on regression statistics proposed by the present invention. DETAILED DESCRIPTION

[0088] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0089] refer to Figure 1-Figure 2A method for detecting and cleaning water level time series data based on regression statistics, characterized in that it comprises the following steps:

[0090] S1. Sort out the historical flood data of the monitoring station, extract the water level time series data of each flood, and perform sliding time series processing on it to form a complete flood sample set;

[0091] In this implementation, S1 specifically includes:

[0092] S11. Set the flood limit water level Z of the monitoring station f , traverse the historical water level time series data of the monitoring station, and start recording the data of this period when the water level sequence shows a monotonically increasing trend;

[0093] S12. Detect temporary peak value Z in water level time series data tmp , when Z tmp >Z f When a flood occurs, it is determined that there is a flood process, and the flood starting point and flood peak point are determined by tracing back from the temporary peak.

[0094] S13, starting from the flood peak point, traverse the water level time series data one by one backward until the water level drops and loses monotonicity, mark the flood end point, and thus extract the complete flood water level time series process;

[0095] S14. For the extracted flood water level time series data, the water level data is processed using a sliding average algorithm to calculate the sliding average of each time series point. The sliding window size is N:

[0096]

[0097] in, is the sliding average of the i-th time point, Z j is the original water level value at the jth time point, N is the sliding window size, and the appropriate N value is set according to the historical flood water level change law;

[0098] S15. After the sliding average processing, a continuous and smooth time-series water level sample set is formed for each flood process.

[0099] S2. According to the nonlinear variation characteristics of flood water level, a regression function cluster is established, including quadratic curve, exponential curve, modified exponential curve, CompertZ curve and Logistic curve;

[0100] In this implementation, S2 specifically includes:

[0101] S21. Analyze the nonlinear variation of flood water levels, determine the characteristics of flood water levels in the rising and falling stages, including the acceleration and deceleration trends of water level changes, and form a function cluster demand analysis;

[0102] S22, establish a cluster of regression functions, including quadratic curve, exponential curve, modified exponential curve, CompertZ curve and Logistic curve;

[0103] S23. When establishing a regression function cluster, the fitting effect of historical flood data is analyzed, the function types that are not suitable for describing the flood process are eliminated, and the composition of the function cluster is dynamically adjusted so that the composition of the regression function cluster can cover the diversity of flood water level changes.

[0104] S3, fitting the historical flood sample data with a regression function, using the curve linearization method to calculate the linear correlation between each fitting function and the actual data, and the regression function with the highest correlation is taken as the optimal regression function;

[0105] In this implementation, S3 specifically includes:

[0106] S31, select the water level data of the first N time series points from the historical flood sample data to form a time series data sequence {Z1, Z2, ..., Z N};

[0107] S32, for each regression function in the regression function cluster, use the linearization method to transform the time series data sequence {Z1, Z2, …, Z N} to convert and obtain the corresponding linearized data sequence;

[0108] S33. Calculate the linear correlation between the original time series data sequence and each regression function linearized data sequence:

[0109]

[0110] Among them, R xy is the linear correlation coefficient, x i and i are the values ​​of the time series data points before and after linearization, and x i and i The mean of

[0111] S34, comparing the linear correlation coefficients of different regression functions, selecting the regression function corresponding to the maximum absolute value of the correlation coefficient as the optimal regression function of the current time series point, repeating the above calculation process point by point for the fitting process of the entire time series data, recording the optimal regression function corresponding to each time series point, and generating an optimal regression function sequence;

[0112] S35. Based on the generated optimal regression function sequence, the prediction value of each time series point is calculated by combining the original time series data and the fitting result of the optimal regression function sequence.

[0113]

[0114] in, is the predicted value of the i-th time point, f(x i ; θ) is the optimal regression function corresponding to the i-th time point, and θ is the parameter of the optimal regression function.

[0115] S4. Based on the fitting result of the optimal regression function, the difference between the predicted value and the measured value is calculated, and a statistical analysis is performed on the difference to establish a Gaussian distribution model of the difference;

[0116] In this implementation, S4 specifically includes:

[0117] S41. Prediction value for each time point in historical flood sample data and the measured value Z i Perform difference calculation:

[0118]

[0119] Among them, ΔZ i is the difference of the i-th time point, is the predicted value calculated according to the optimal regression function, Z i is the actual measured value;

[0120] S42, for all the differences {ΔZ1, ΔZ2, ..., ΔZ N}, use statistical methods to calculate its mean μ and variance σ of the difference 2 :

[0121]

[0122] Among them, N is the total number of difference sample points, μ is the mean of the difference, σ 2 is the variance of the difference;

[0123] S43. Based on the mean and variance of the difference, a Gaussian distribution model is constructed:

[0124]

[0125] Among them, p(x) is the probability density of Gaussian distribution at point x, μ is the mean of the difference, σ 2 is the variance of the difference;

[0126] S44. According to the Gaussian distribution model, the range of the difference distribution is calculated by verifying the sample and the confidence interval [-v,v] is determined:

[0127] v = μ ± kσ;

[0128] Wherein, v is the upper limit of the difference range that meets a certain confidence level, and k is the confidence factor, which is determined according to the set confidence level. The value of k=1.96 corresponds to a confidence level of 95%.

[0129] S5. Determine the confidence interval of the Gaussian distribution through the verification sample, and determine whether the data is abnormal data based on the confidence interval;

[0130] In this implementation, S5 specifically includes:

[0131] S51. Based on the Gaussian distribution model and the confidence interval of the difference [-v, v], each time series point in the flood time series data is tested to determine the predicted value of the current time series point. and the measured value Z i The difference ΔZ i Whether it exceeds the confidence interval range;

[0132] S52, when ΔZ i Satisfy v≤ΔZ i ≤v, the data at the current timing point is considered normal. i If the above conditions are not met, that is, ΔZi<-v and ΔZi>v, the current time sequence point is determined to be abnormal data, and the time sequence number of the abnormal point and its corresponding difference Z are recorded. i ;

[0133] S53: For the detected abnormal data, mark the current time series point data as abnormal, retain the upper and lower adjacent data points of the flood time series, and record the measurement value Z of the abnormal data point. i and the corresponding predicted value

[0134]

[0135] S54, for each abnormal data point, the difference ΔZ i Perform classification statistics to generate abnormal data distribution characteristics, including the mean and variance of abnormal point difference values ​​and the distribution of offsets outside the confidence interval;

[0136] S56, separating the detected abnormal data from the normal data to form two independent data sets, namely the normal data set Z normal and the abnormal data set Z anomaly .

[0137] S6. For the detected abnormal data, according to the regression function parameters and the Euclidean distance calculation, the regression function with the smallest historical fitting parameter distance is selected to correct the abnormal data to obtain the cleaned data;

[0138] In this implementation, S6 specifically includes:

[0139] S61, for the detected abnormal data point Z anomaly , extract the corresponding optimal regression function parameter set θ i , where θ i =(θ i1 ,θ i2 ,,θ in ) represents the regression function parameter of the i-th abnormal data point;

[0140] S62. Extract the historical parameter set of the same function type from the historical regression fitting results

[0141] {θ h1 ,θ h2 ,...,θ hm}, where m is the number of functions of the same type in the history matching results;

[0142] S63, calculate the regression parameter θ of the abnormal data point i With the historical parameter set θ hj The Euclidean distance d j :

[0143]

[0144] Among them, θ ik and θ hjk are the values ​​of the kth parameter in the abnormal data point and the historical fitting result, respectively, and d j is the Euclidean distance between the outlier parameter and the jth historical parameter;

[0145] S64. Among all historical parameters, find the parameter θ with the smallest Euclidean distance hmin , and its corresponding historical fitting prediction value is f(x i θ hmin ), where f(x i ; θ) represents the predicted value of the optimal regression function;

[0146] S65, predicting the value based on the current regression function of the abnormal data point The predicted value f(x i θ hmin ), correct the measured values ​​of abnormal data points:

[0147] Z cleaned =f(x i θ hmin );

[0148] Among them, Z cleaned is the cleaned value corrected for abnormal data points;

[0149] S66, the cleaned data Z cleanedReplace the original abnormal data Z anomaly , while retaining the regression function parameters and calculation details in the correction process, verification and model optimization, repeating the correction process for all detected abnormal data points until the entire flood time series data is cleaned and the final cleaned time series data set Z is generated final .

[0150] S7. The first N time series points in the flood time series data are processed using a sliding average algorithm to eliminate the interference of accidental fluctuations on regression fitting.

[0151] In this implementation, S7 specifically includes:

[0152] S71. For flood time series data {Z1, Z2, ..., Z N} perform sliding average processing on the first M time points in the sequence, and calculate the sliding average value of each time point

[0153] S72, setting a sliding window size k, and determining a value range of k according to the fluctuation characteristics of historical flood water level data;

[0154] S73, for the data points {Z1, Z2, ..., Z k}, complete the window by repeatedly expanding the existing data points and calculate the sliding average of the initial time series points:

[0155]

[0156] Where i is the number of the current time series point, and j is the number of the water level data point in the starting window;

[0157] S74. After the sliding average processing is completed, a smoothed water level data sequence is generated. During the sliding average processing, the processed data sequence is detected in real time Is there a mutation? If abnormal fluctuations are detected, record the abnormal points and keep the original data Z i .

[0158] Embodiment 1:

[0159] In order to verify the feasibility of the present invention, the present invention is applied to the flood monitoring and data cleaning work of a large reservoir in Sichuan Province in July 2023. The reservoir is located in the mountainous area of ​​western Sichuan. The river network in the basin is dense, the rainfall intensity during the flood season is large and unevenly distributed, and flash floods and flood superpositions often occur. Because the sensors in the flood monitoring system have been operating in a complex natural environment for a long time, they are often affected by external interference such as heavy rain, strong winds, and silt scouring, resulting in frequent abnormal points in the collected water level time series data. These abnormal data may cause prediction errors in the flood forecasting model, affecting the scientificity and timeliness of flood control command. Therefore, the reservoir management department decided to adopt the water level time series data detection and cleaning method based on regression statistics of the present invention to improve the quality and accuracy of flood forecast data.

[0160] In this scenario, the application of the present invention is divided into two stages: historical data modeling and real-time monitoring and processing. In the historical data modeling stage, the flood water level data of the reservoir in the past five years were first systematically sorted, and a total of 30 time series data of complete flood processes were extracted as modeling samples. These data are used to construct a regression function cluster including quadratic curves, exponential curves, modified exponential curves, CompertZ curves and Logistic curves. In order to capture the nonlinear variation law of flood water levels, the system performs regression fitting on the rising stage, peak stage and falling stage of each flood, selects the optimal regression function through the linearization method, and records the corresponding regression parameters and fitting errors. Subsequently, according to the fitting results, the difference between the predicted value and the actual measured value is calculated, a Gaussian distribution model of the difference is established, and the confidence interval of the difference is determined by verifying the sample data.

[0161] In the real-time monitoring and processing stage, the present invention is used for the data cleaning of the continuous flood process from July 1 to July 15, 2023. First, the system performs sliding average preprocessing on the real-time collected water level data to smooth out short-term fluctuations in the data. The sliding window size is set to 10 time points, which eliminates the fluctuations caused by the short-term influence of the external environment on the sensor during heavy rain. Subsequently, the system performs anomaly detection on each time point in the water level data based on the constructed Gaussian distribution model and confidence interval. If the difference between the predicted value and the actual measured value of a time point exceeds the confidence interval range, the system judges the point as an abnormal data point and marks its location and deviation amplitude.

[0162] For the detected abnormal data points, the system uses the Euclidean distance calculation method to find the fitting curve closest to the current regression parameters in the historical fitting parameters, and uses the predicted value of the curve to correct the abnormal points. The corrected cleaned data remains consistent with the surrounding normal data, which not only avoids the data loss problem caused by simply eliminating abnormal points in traditional methods, but also ensures the integrity and coherence of time series data.

[0163] In this flood monitoring work, the system processed a total of 36,000 time series data points, of which 512 abnormal data points were detected, accounting for 1.42% of the total data volume. After cleaning, the proportion of abnormal data points was reduced to 0.02%, and the average absolute error was reduced from ±5.3cm before cleaning to ±1.2cm. In actual flood forecasting, the prediction error of the peak arrival time was ±4 hours before cleaning, and it was reduced to ±1 hour after cleaning; the peak water level prediction error was reduced from ±12cm to ±3cm.

[0164] In addition, the present invention demonstrates high efficiency in real-time processing. The system completes anomaly detection and cleaning in 3.8 seconds per 100 time points in a single run, fully meeting the needs of real-time flood monitoring. During a heavy rainfall on July 10, the system processed the rapidly growing water level data in a short period of time in real time, effectively detected and corrected multiple anomalies, enabling reservoir management departments to accurately predict the arrival time of the flood peak and water level changes, and gaining valuable time for timely dispatch of flood control measures.

[0165] Table 1 Comparison of flood level data quality and prediction performance before and after cleaning

[0166] Indicator Category Before cleaning After cleaning The proportion of abnormal data points 1.42％ 0.02％ Mean absolute error ±5.3cm ±1.2cm Peak arrival time error ±4 hours ±1 hour Flood peak water level prediction error ±12cm ±3cm Data processing time efficiency Unable to process in real time 3.8 seconds / 100 timing points

[0167] It can be seen from the above table that in practical applications, the effect of the present invention significantly improves the quality of flood water level time series data and the accuracy of flood forecasting models. Before the deployment of the present invention, the proportion of abnormal data points in the water level data was relatively high, reaching 1.42%. After cleaning, the proportion of abnormal data points dropped to 0.02%, which basically eliminated the impact of abnormal data on the flood forecasting model. At the same time, the mean absolute error of the water level data was significantly reduced from ±5.3cm before cleaning to ±1.2cm, the prediction error of the peak arrival time was shortened from ±4 hours to ±1 hour, and the peak water level prediction error was reduced from ±12cm to ±3cm. These improvements have greatly improved the reliability and accuracy of flood forecasting.

[0168] The time efficiency of the present invention has also been verified in practical applications. Before cleaning, traditional methods cannot meet the needs of real-time flood monitoring and data cleaning due to their reliance on manual processing and static rules. The present invention, through an automated anomaly detection and correction mechanism, can process data from 100 time series points within 3.8 seconds, achieving real-time processing and efficient response. During the heavy rainfall on July 10, 2023, the system quickly detected and corrected multiple abnormal data points caused by interference with the sensors, providing accurate flood peak forecasts for reservoir management departments, ensuring the timeliness and scientificity of flood control scheduling decisions.

[0169] Through the application of the present invention, the reservoir management department avoided multiple flood forecast deviations caused by abnormal data during the entire monitoring period. In a specific case, the system detected and corrected an abnormal data point in the rising stage of the flood. This data point may cause the predicted flood peak time to be advanced by 4 hours in traditional methods. However, the present invention controlled the prediction error within 1 hour through precise correction, successfully ensuring the safe transfer of downstream residents.

[0170] The above data and application scenarios fully demonstrate that the present invention not only has significant advantages in detecting and correcting abnormal data, but also can improve the overall performance of the flood monitoring system in an efficient and accurate manner, providing a reliable data basis for flood forecasting in complex environments. These results show that the application of the present invention in the field of water conservancy has broad prospects and practical value.

[0171] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent substitutions and changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for detecting and cleaning water level time series data based on regression statistics, characterized in that: The following steps are involved: S1. Sort out the historical flood data of the monitoring station, extract the water level time series data of each flood, and perform sliding time series processing on it to form a complete flood sample set; S2. According to the nonlinear variation characteristics of flood water level, a regression function cluster is established, including quadratic curve, exponential curve, modified exponential curve, CompertZ curve and Logistic curve; S3, fitting the historical flood sample data with a regression function, using the curve linearization method to calculate the linear correlation between each fitting function and the actual data, and the regression function with the highest correlation is taken as the optimal regression function; S4. Based on the fitting result of the optimal regression function, the difference between the predicted value and the measured value is calculated, and a statistical analysis is performed on the difference to establish a Gaussian distribution model of the difference; S5. Determine the confidence interval of the Gaussian distribution through the verification sample, and determine whether the data is abnormal data based on the confidence interval; S6. For the detected abnormal data, according to the regression function parameters and the Euclidean distance calculation, the regression function with the smallest historical fitting parameter distance is selected to correct the abnormal data to obtain the cleaned data; S7. The first N time series points in the flood time series data are processed using a sliding average algorithm to eliminate the interference of accidental fluctuations on regression fitting.

2. According to the method for detecting and cleaning water level time series data based on regression statistics in claim 1, it is characterized in that: The S1 specifically includes: S11. Set the flood limit water level Z of the monitoring station f , traverse the historical water level time series data of the monitoring station, and start recording the data of this period when the water level sequence shows a monotonically increasing trend; S12. Detect temporary peak value Z in water level time series data tmp , when Z tmp >Z f When a flood occurs, it is determined that there is a flood process, and the flood starting point and flood peak point are determined by tracing back from the temporary peak. S13, starting from the flood peak point, traverse the water level time series data one by one backward until the water level drops and loses monotonicity, mark the flood end point, and thus extract the complete flood water level time series process; S14. For the extracted flood water level time series data, the water level data is processed using a sliding average algorithm to calculate the sliding average of each time series point. The sliding window size is N: in, is the sliding average of the i-th time point, Z j is the original water level value at the jth time point, N is the sliding window size, and the appropriate N value is set according to the historical flood water level change law; S15. After the sliding average processing, a continuous and smooth time-series water level sample set is formed for each flood process.

3. The method for detecting and cleaning water level time series data based on regression statistics according to claim 1 is characterized in that: The S2 specifically includes: S21. Analyze the nonlinear variation of flood water levels, determine the characteristics of flood water levels in the rising and falling stages, including the acceleration and deceleration trends of water level changes, and form a function cluster demand analysis; S22, establish a cluster of regression functions, including quadratic curve, exponential curve, modified exponential curve, CompertZ curve and Logistic curve; S23. When establishing a regression function cluster, the fitting effect of historical flood data is analyzed, the function types that are not suitable for describing the flood process are eliminated, and the composition of the function cluster is dynamically adjusted so that the composition of the regression function cluster can cover the diversity of flood water level changes.

4. The method for detecting and cleaning water level time series data based on regression statistics according to claim 1 is characterized in that: The S3 specifically includes: S31, select the water level data of the first N time series points from the historical flood sample data to form a time series data sequence {Z1, Z2, ..., Z N }; S32, for each regression function in the regression function cluster, use the linearization method to transform the time series data sequence {Z1, Z2, …, Z N } to convert and obtain the corresponding linearized data sequence; S33. Calculate the linear correlation between the original time series data sequence and each regression function linearized data sequence: Among them, R xy is the linear correlation coefficient, x i and i are the values ​​of the time series data points before and after linearization, and x i and i The mean of S34, comparing the linear correlation coefficients of different regression functions, selecting the regression function corresponding to the maximum absolute value of the correlation coefficient as the optimal regression function of the current time series point, repeating the above calculation process point by point for the fitting process of the entire time series data, recording the optimal regression function corresponding to each time series point, and generating an optimal regression function sequence; S35. Based on the generated optimal regression function sequence, the prediction value of each time series point is calculated by combining the original time series data and the fitting result of the optimal regression function sequence. in, is the predicted value of the i-th time point, f(x i ; θ) is the optimal regression function corresponding to the i-th time point, and θ is the parameter of the optimal regression function.

5. The method for detecting and cleaning water level time series data based on regression statistics according to claim 1 is characterized in that: The S4 specifically includes: S41. Prediction value for each time point in historical flood sample data and the measured value Z i Perform difference calculation: Among them, ΔZ i is the difference of the i-th time point, is the predicted value calculated according to the optimal regression function, Z i is the actual measured value; S42, for all the differences {ΔZ1, ΔZ2, ..., ΔZ N }, use statistical methods to calculate its mean μ and variance σ of the difference 2 : Among them, N is the total number of difference sample points, μ is the mean of the difference, σ 2 is the variance of the difference; S43. Based on the mean and variance of the difference, a Gaussian distribution model is constructed: Among them, p(x) is the probability density of Gaussian distribution at point x, μ is the mean of the difference, σ 2 is the variance of the difference; S44. According to the Gaussian distribution model, the range of the difference distribution is calculated by verifying the sample and the confidence interval 9-v,v3 is determined: v = μ ± kσ; Wherein, v is the upper limit of the difference range that meets a certain confidence level, and k is the confidence factor, which is determined according to the set confidence level. The value of k=1.96 corresponds to a confidence level of 95%.

6. The method for detecting and cleaning water level time series data based on regression statistics according to claim 1 is characterized in that: The S5 specifically includes: S51. Based on the Gaussian distribution model and the confidence interval of the difference [-v, v], each time series point in the flood time series data is tested to determine the predicted value of the current time series point. and the measured value Z i The difference ΔZ i Whether it exceeds the confidence interval range; S52, when ΔZ i Satisfy v≤ΔZ i ≤v, the data at the current timing point is considered normal. i If the above conditions are not met, that is, ΔZi<-v and ΔZi>v, the current time sequence point is determined to be abnormal data, and the time sequence number of the abnormal point and its corresponding difference Z are recorded. i ; S53: For the detected abnormal data, mark the current time series point data as abnormal, retain the upper and lower adjacent data points of the flood time series, and record the measurement value Z of the abnormal data point. i and the corresponding predicted value S54, for each abnormal data point, the difference ΔZ i Perform classification statistics to generate abnormal data distribution characteristics, including the mean and variance of abnormal point difference values ​​and the distribution of offsets outside the confidence interval; S56, separating the detected abnormal data from the normal data to form two independent data sets, namely the normal data set Z normal and the abnormal data set Z anomaly .

7. The method for detecting and cleaning water level time series data based on regression statistics according to claim 1 is characterized in that: The S6 specifically includes: S61, for the detected abnormal data point Z anomaly , extract the corresponding optimal regression function parameter set θ i , where θ i =(θ i1 ,θ i2 ,,θ in ) represents the regression function parameter of the i-th abnormal data point; S62, extract the historical parameter set {θ h1 ,θ h2 ,...,θ hm }, where m is the number of functions of the same type in the history matching results; S63, calculate the regression parameter θ of the abnormal data point i With the historical parameter set θ hj The Euclidean distance d j : Among them, θ ik and θ hjk are the values ​​of the kth parameter in the abnormal data point and the historical fitting result, respectively, and d j is the Euclidean distance between the outlier parameter and the jth historical parameter; S64. Among all historical parameters, find the parameter θ with the smallest Euclidean distance hmin , and its corresponding historical fitting prediction value is f(x i θ hmin ), where f(x i ; θ) represents the predicted value of the optimal regression function; S65, predicting the value based on the current regression function of the abnormal data point The predicted value f(x i θ hmin ), correct the measured values ​​of abnormal data points: Z cleaned =f(x i ;θ hmin ); Among them, Z cleaned is the cleaned value corrected for abnormal data points; S66, the cleaned data Z cleaned Replace the original abnormal data Z anomaly , while retaining the regression function parameters and calculation details in the correction process, verification and model optimization, repeating the correction process for all detected abnormal data points until the entire flood time series data is cleaned and the final cleaned time series data set Z is generated final .

8. The method for detecting and cleaning water level time series data based on regression statistics according to claim 1 is characterized in that: The S7 specifically includes: S71. For flood time series data {Z1, Z2, ..., Z N } perform sliding average processing on the first M time points in the sequence, and calculate the sliding average value of each time point S72, setting a sliding window size k, and determining a value range of k according to the fluctuation characteristics of historical flood water level data; S73, for the data points {Z1, Z2, ..., Z k }, complete the window by repeatedly expanding the existing data points and calculate the sliding average of the initial time series points: Where i is the number of the current time series point, and j is the number of the water level data point in the starting window; S74. After the sliding average processing is completed, a smoothed water level data sequence is generated. During the sliding average processing, the processed data sequence is detected in real time Is there a mutation? If abnormal fluctuations are detected, record the abnormal points and keep the original data Z i .

Images