A hydrological real-time sequence abnormal value detection method and system based on R-ARIMA and fluctuation test

Abstract

The application discloses a hydrological real-time sequence anomaly value detection method and system based on R-ARIMA and fluctuation testing. The method balances the risks of misjudgment and missed judgment by comprehensively testing whether there is a significant difference between the regularity and fluctuation of the to-be-detected data and the hydrological quantity time sequence, and has good robustness and accuracy under both multi-time scale periods and local severe fluctuation working conditions. Meanwhile, the self-iteration mechanism also guarantees the long-term stability of the application. Through example simulation, it is shown that the application has high actual precision and light load compared with other algorithms.

Description

Technical Field

[0001] This invention belongs to the field of hydrological monitoring and data processing technology, and in particular relates to a method and system for detecting outliers in real-time hydrological sequences based on R-ARIMA and fluctuation test. Background Technology

[0002] Hydrological time series are sequences of observed values ​​(such as flow, water level, and rainfall) arranged chronologically. However, the presence of outliers in these sequences can lead to misjudgments or false alarms in online monitoring, and can also severely interfere with model generation, forecasting, and early warning in time series analysis, distorting the results of hydrological pattern analysis. Therefore, real-time and accurate detection of outliers in hydrological sequences is of great practical significance for improving the operational efficiency of water conservancy and hydropower units and enhancing the accuracy of hydrological analysis and modeling.

[0003] The causes of outliers in hydrological time series are complex, mainly including: errors in data measurement, transmission, and entry; series jumps caused by sudden changes in environmental factors (such as gate opening and closing, heavy rainfall, and engineering construction); and data inconsistencies caused by changes in measurement methods or benchmarks. Therefore, effectively detecting outliers helps to uncover hidden information and patterns behind the series, which is crucial for supporting decision-making.

[0004] Existing online methods for detecting outliers in real-time hydrological sequences include model prediction, hypothesis testing, clustering, classification, time-frequency analysis, Monte Carlo methods, correlation analysis, and principal hierarchical analysis. These methods differ in their mechanisms, prerequisites, and applicable scenarios, such as specific requirements regarding the ergodicity of the data, the completeness of influencing factors, or the model structure. Furthermore, relying on a single method inevitably has limitations, such as a high risk of misjudgment or missed detection, or insufficient robustness under specific conditions. For example, simple statistical threshold methods (such as 3...) The criteria are prone to misjudging normal extreme values ​​as abnormalities when the sequence fluctuates drastically; while complex machine learning methods (such as clustering and classification) often have a large computational load and are difficult to meet the needs of real-time iterative updates. Summary of the Invention

[0005] Purpose of the invention: This invention provides a method and system for detecting outliers in real-time hydrological sequences based on R-ARIMA and fluctuation tests. It aims to solve the problems of existing technologies, such as strong dependence on data characteristics, high risk of misjudgment and missed judgment, and complex calculations that make real-time iteration difficult. By integrating time series prediction and statistical fluctuation tests and introducing a self-iterative update mechanism, it achieves real-time anomaly detection with high accuracy, low load, strong comprehensiveness, high robustness, lightweight calculation, and long-term stable operation.

[0006] Technical solution: The present invention provides a method for detecting outliers in real-time hydrological sequences based on R-ARIMA and fluctuation tests, comprising:

[0007] S1. Obtain a preset number of hydrological measurements to form the most recent sequencing sequence, and construct an ARIMA model based on the most recent sequencing sequence.

[0008] S2. Receive the latest measured value and use the ARIMA model to generate a predicted value. Perform Grubbs difference verification on the predicted value and the latest measured value. If the verification passes, the latest measured value is determined to be a normal value and proceed to S4. If the verification fails, proceed to S3 for a second verification.

[0009] S3. Perform a Dixon volatility test on the predicted value and the latest measured value. If the test passes, the latest measured value is determined to be a normal value and proceed to S4. If the test fails, the latest measured value is determined to be an abnormal value.

[0010] S4. Add the latest measurement to the recent measurement sequence using a sliding window, and iteratively update the ARIMA model using the residual gradient method.

[0011] Furthermore, the construction of the ARIMA model based on the most recent sequencing sequence includes:

[0012] The ADF test was used to analyze the most recent measurement sequence. Continue performing the difference operation until the most recent measured sequence is reached. Until the condition stabilizes, the difference order is obtained. The stationary sequence after difference;

[0013] Perform a white noise test on the stationary sequence after difference. If the test result is a non-white noise sequence, then obtain a non-white noise measurement sequence; otherwise, re-examine the data or adjust the sequence length.

[0014] Determine the optimal autoregressive order of the non-white noise measurement sequence based on the Bayesian information criterion. and optimal moving average order ;

[0015] Based on certainty The parameters are set, an ARIMA model is built using maximum likelihood estimation, and the residuals of the ARIMA model are subjected to a white noise test. If the test passes, the ARIMA model is output; otherwise, the parameters are adjusted. .

[0016] Furthermore, the Grubbs difference verification includes:

[0017] Constructing the Grubbs difference test statistic The formula is:

[0018] ;

[0019] in, The most recent measurement sequence standard deviation For predicted values, These are the latest measurements;

[0020] At the preset significance level Below, if the difference test statistic Greater than or equal to the preset Grubbs threshold , The most recent measurement sequence The number of measurements, i.e. Then determine the latest measured value. Compared with the predicted value If a significant difference exists, proceed to S3 for a secondary test; if the difference test statistic... Less than Grubbs threshold ,Right now Then determine the latest measured value. If the value is normal, proceed to S4.

[0021] Furthermore, the Dixon volatility test includes:

[0022] Construct the Dixon volatility test statistic The formula is:

[0023] ;

[0024] in, These are the latest measurements. The nth measurement in the most recent measurement sequence, For the (n-1)th measurement in the most recent measurement sequence, The most recent measurement sequence The maximum value among the differences between two consecutive terms. The most recent measurement sequence The minimum value among two consecutive differences;

[0025] At the preset significance level If the volatility test statistic is below Greater than or equal to the pre-set critical value of the Dixon test ,Right now The latest measured value Volatility and recent measurement series Significant inconsistency, determine the latest measurement. It is an outlier; if the volatility test statistic is... Less than the critical value of the Dixon test ,Right now If the latest measured value is determined to be normal, proceed to S4.

[0026] Furthermore, the residual gradient method is used to iteratively update the ARIMA model, as shown in the formula:

[0027] ;

[0028] in, These are the updated ARIMA model coefficients. These are the coefficients of the ARIMA model before the update. For dynamic learning rate, loss function The gradient of the model coefficients, the loss function The prediction residual is determined by the difference between the latest measured value and the predicted value; the dynamic learning rate is based on the sigmoid function and decays with the number of iterations.

[0029] The present invention discloses a real-time hydrological series outlier detection system based on R-ARIMA and fluctuation test, comprising:

[0030] The data acquisition and model building module is used to acquire a preset number of hydrological measurements to form the most recent sequencing sequence, and to build an ARIMA model based on the most recent sequencing sequence.

[0031] The difference verification module is used to receive the latest measured value and generate a predicted value using the ARIMA model. It performs Grubbs difference verification on the predicted value and the latest measured value. If the verification passes, the latest measured value is determined to be a normal value and enters the model update module. If the verification fails, it enters the volatility verification module for a second verification.

[0032] The volatility test module is used to perform a Dixon volatility test on the predicted value and the latest measured value. If the test passes, the latest measured value is determined to be a normal value and the module will proceed to update the model. If the test fails, the latest measured value is determined to be an outlier.

[0033] The model update module is used to add the latest measurement to the recent measurement sequence in a sliding window manner, and to iteratively update the ARIMA model using the residual gradient method.

[0034] Furthermore, in the data acquisition and model building module, the construction of the ARIMA model based on the most recent sequencing sequence includes:

[0035] The ADF test was used to analyze the most recent measurement sequence. Continue performing the difference operation until the most recent measured sequence is reached. Until the condition stabilizes, the difference order is obtained. The stationary sequence after difference;

[0036] Perform a white noise test on the stationary sequence after difference. If the test result is a non-white noise sequence, then obtain a non-white noise measurement sequence; otherwise, re-examine the data or adjust the sequence length.

[0037] Determine the optimal autoregressive order of the non-white noise measurement sequence based on the Bayesian information criterion. and optimal moving average order ;

[0038] Based on certainty The parameters are set, an ARIMA model is built using maximum likelihood estimation, and the residuals of the ARIMA model are subjected to a white noise test. If the test passes, the ARIMA model is output; otherwise, the parameters are adjusted. .

[0039] Furthermore, in the difference verification module, the Grubbs difference verification includes:

[0040] Constructing the Grubbs difference test statistic The formula is:

[0041] ;

[0042] in, The most recent measurement sequence standard deviation For predicted values, These are the latest measurements;

[0043] At the preset significance level Below, if the difference test statistic Greater than or equal to the preset Grubbs threshold , The most recent measurement sequence The number of measurements, i.e. Then determine the latest measured value. Compared with the predicted value If a significant difference exists, proceed to the volatility test module for a secondary test; if the difference test statistic... Less than Grubbs threshold ,Right now Then determine the latest measured value. If the value is normal, proceed to the model update module.

[0044] Furthermore, in the volatility testing module, the Dixon volatility test includes:

[0045] Construct the Dixon volatility test statistic The formula is:

[0046] ;

[0047] in, These are the latest measurements. The nth measurement in the most recent measurement sequence, For the (n-1)th measurement in the most recent measurement sequence, The most recent measurement sequence The maximum value among the differences between two consecutive terms. The most recent measurement sequence The minimum value among two consecutive differences;

[0048] At the preset significance level If the volatility test statistic is below Greater than or equal to the pre-set critical value of the Dixon test ,Right now The latest measured value Volatility and recent measurement series Significant inconsistency, determine the latest measurement. It is an outlier; if the volatility test statistic is... Less than the critical value of the Dixon test ,Right now If the latest measured value is found to be normal, the system will proceed to the model update module.

[0049] Furthermore, in the model update module, the residual gradient method is used to iteratively update the ARIMA model, and the formula is:

[0050] ;

[0051] in, These are the updated ARIMA model coefficients. These are the coefficients of the ARIMA model before the update. For dynamic learning rate, loss function The gradient of the model coefficients, the loss function The prediction residual is determined by the difference between the latest measured value and the predicted value; the dynamic learning rate is based on the sigmoid function and decays with the number of iterations.

[0052] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0053] 1. High detection accuracy and low risk of false positives and false negatives: Through a dual testing mechanism of regularity test (Grubbs criterion) and volatility test (Dixon criterion), it considers both the long-term trend of the sequence and reasonable local fluctuations, distinguishes between true outliers and normal extreme values, and improves recall and precision.

[0054] 2. Strong robustness: This invention does not rely on a single statistical characteristic and can maintain stable detection performance under complex hydrological conditions with multiple time scale periodic changes or severe local fluctuations (such as typhoons and floods).

[0055] 3. Computationally lightweight and suitable for real-time processing: The core algorithm is based on the classic ARIMA model and statistical tests, resulting in low computational complexity. At the same time, combined with the efficient residual gradient update method, the model iterates and updates quickly, which can meet the real-time online detection requirements of high-frequency hydrological data acquisition (such as every 5 minutes).

[0056] 4. Good long-term stability: Through iterative updates, the ARIMA model can continuously adapt to the non-stationary and nonlinear dynamic changes of hydrological sequences, avoiding the decline in detection performance due to model aging and ensuring the long-term reliable operation of the method. Attached Figure Description

[0057] Figure 1 This is a flowchart of the method of the present invention.

[0058] Figure 2 This is a detailed logic flowchart of the present invention. Detailed Implementation

[0059] Example 1

[0060] like Figure 1 and Figure 2 As shown, the present invention provides a method for detecting outliers in real-time hydrological sequences based on R-ARIMA and fluctuation tests, comprising:

[0061] S1. Obtain a preset number of hydrological measurements to form the most recent sequencing sequence, and calculate the standard deviation of the most recent sequencing sequence; construct an ARIMA model based on the most recent sequencing sequence;

[0062] Specifically, S1 includes:

[0063] S1.1. Extract a specified number of hydrological monitoring values ​​(e.g., the most recent 72) in ascending order of time to form a recent measurement sequence, denoted as... And calculate the standard deviation of the sequence. ;

[0064] S1.2, For the most recent measurement sequence Establishing an ARIMA (Autoregressive Integral Moving Average) model includes:

[0065] S1.2.1, Sequence Preprocessing and Stationarity Test: The ADF (Augmented Dickey-Fuller) test is used as the algorithm for testing the stationarity of the time series. This is achieved by analyzing the most recent measured sequence... Continue performing differencing operations. If the ADF test statistic is less than the preset stationarity test critical value, then the most recent measurement sequence is determined. If the sequence is stable, continue differencing until the most recent measurement sequence is reached. Until the condition stabilizes, the difference order is obtained. The stationary sequence is then differiated. Subsequently, the Ljung-BoxQ test is used as the white noise test algorithm to test the stationary sequence after difference. If the test result is a non-white noise sequence, it indicates that there is effective information in the stationary sequence after difference that can be used for modeling, and a non-white noise measurement sequence is obtained, and the next step is carried out. If the stationary sequence after difference is white noise, the data needs to be re-examined or the sequence length needs to be adjusted.

[0066] S1.2.2 Model Order Determination: Based on the Bayesian Information Criterion (BIC), determine the optimal autoregressive order for the non-white noise measurement sequence. and optimal moving average order ;

[0067] S1.2.3 Model Validation: Based on deterministic... The parameters are used to build an ARIMA model using maximum likelihood estimation, and the residuals of the ARIMA model are tested for white noise. If the test passes, the ARIMA model is output for prediction; otherwise, the parameters are adjusted. ;

[0068] S2, Receive the latest measurement value The ARIMA model is used to generate predicted values. For the predicted value Compared with the latest measurement Perform a Grubbs difference check; if the check passes, the latest measurement is determined. If the value is within the normal range, proceed to step S4. If the test fails, the latest measured value is then determined. Compared with the predicted value There is a significant difference, so proceed to S3 for a second test;

[0069] Specifically, S2 includes:

[0070] S2.1 Receive the latest measurement value The ARIMA model is used to generate predicted values. ;

[0071] S2.2. Use the Grubbs criterion to verify the latest measured values. Compared with the predicted value To determine the differences between them, construct the Grubbs difference test statistic. The formula is:

[0072] ;

[0073] in, The most recent measurement sequence Standard deviation;

[0074] At the preset significance level (Usually taken as 0.05), if the difference test statistic... Greater than or equal to the preset Grubbs threshold , The most recent measurement sequence The number of measurements, i.e. Then determine the latest measured value. Compared with the predicted value If a significant difference exists, proceed to S3 for a secondary test; if the difference test statistic... Less than Grubbs threshold ,Right now Then determine the latest measured value. If the value is normal, proceed to S4;

[0075] S3, For predicted values Compared with the latest measurement Perform a Dixon volatility test; if the test passes, the latest measurement is considered valid. If the value is within the normal range, proceed to step S4. If the test fails, the latest measured value is then determined. This is an outlier;

[0076] Specifically, if the Grubbs difference test fails, it indicates that the latest measurement may deviate from the long-term trend of the sequence. To distinguish between a genuine anomaly and a reasonable localized sharp fluctuation (such as a flood), a volatility test is required, S3 of which includes:

[0077] The latest measured value was determined using the Dixon test criterion. Does it belong to the most recent measurement sequence? To determine the reasonable range of fluctuations, construct the Dixon volatility test statistic. The formula is:

[0078] ;

[0079] These are the latest measurements. The nth measurement in the most recent measurement sequence, For the (n-1)th measurement in the most recent measurement sequence, The most recent measurement sequence The maximum value among the differences between two consecutive terms. The most recent measurement sequence The minimum value among two consecutive differences;

[0080] At the same significance level If the volatility test statistic is below Greater than or equal to the pre-set critical value of the Dixon test (This was obtained from the Dixon discard quotient critical value table), that is... The latest measured value Volatility and recent measurement series Significant inconsistency, determine the latest measurement. This is an outlier; we await processing the next new measurement. If the volatility test statistic... Less than the critical value of the Dixon test ,Right now This indicates the latest measured value. Although it deviates from the trend, it is within a reasonable fluctuation and should be judged as a normal value, proceeding to S4;

[0081] S4. Display the latest measurement value using a sliding window. Add to recent measurement sequence In the middle, the standard deviation of the most recent measurement sequence is recalculated, and the ARIMA model is updated online using the residual gradient method; return to S2 to receive the new latest measurement and verify it.

[0082] Specifically, S4 includes:

[0083] S4.1, Display the latest measurement value using a sliding window. Add to recent measurement sequence In this approach, the oldest data point is removed, the latest measurement is added to maintain a fixed length, and the standard deviation of the most recent measurement sequence is recalculated. ;

[0084] S4.2. The ARIMA model is updated online using the residual gradient method. The update formula is as follows:

[0085] ;

[0086] in, These are the updated ARIMA model coefficients. These are the coefficients of the ARIMA model before the update. For dynamic learning rate, loss function The gradient of the model coefficients, the loss function From the predicted residual Sure, The prediction residual is the difference between the latest measured value and the predicted value, i.e. ;

[0087] The dynamic learning rate is based on the sigmoid function and decays with the number of iterations.

[0088] S4.3 Return to S2, receive the new latest measurement value and verify it.

[0089] Regarding the above technical solution in this embodiment, the process of applying the technical solution to a specific experiment and the technical effect of the technical solution will be described below.

[0090] Taking the measured flow rate of the Yubojiang Flow Station in a river area of ​​the eastern Zhejiang Plain from 8:00 AM on September 12, 2022 to 8:00 AM on September 19, 2022 as an example, the area experienced heavy rainfall brought by Typhoon Muifa, with the 72-hour average rainfall exceeding 350 mm. The flow station measured the cross-sectional flow rate every 5 minutes, resulting in a total of 2009 raw real-time flow data points. After verification and processing after the flood season, the statistics are as follows: 6 real-time data points were missing due to communication failures, etc., and were supplemented using linear interpolation; a total of 17 raw flow data points were marked as abnormal and were not covered using interpolation or other methods.

[0091] Now, respectively, 3 The method employs the rule of law, Markov model, density-based clustering, and the method of this invention to simulate online outlier detection of the aforementioned original measured flow sequences. Specifically, the Markov model method, based on the information gain principle, divides the flow value range into six equal state intervals; in the clustering method, the neighborhood distance threshold is set to 5% of the maximum flow value, and the neighborhood sample number threshold is set to 1% of the total sample number.

[0092] The parameters were uniformly set to a sliding window sequence length of 72 and a confidence level of 95%.

[0093] The test results obtained by comparison are shown in Table 1.

[0094] Table 1 Comparison of outlier detection results of four methods

[0095]

[0096] A detailed analysis of the results of each method reveals that, 3 The rule-based approach had relatively high rates of missed and false positives, concentrated during periods of heavy rainfall and runoff. This indicates that relying solely on the distribution of values ​​without considering the regularity of hydrological data easily leads to confusion between true extreme values ​​and outliers. The Markov model method suffers from issues such as the division of state intervals for continuous variables, the proportion of each state in the sample space, and the ergodicity of state transitions within the sample. This results in abrupt changes in the numerical values ​​and amplitudes of hydrological data, while the state transition probability calculation remains reasonable, leading to significant missed positives. While the clustering method has high overall detection accuracy, its computational complexity is enormous and it is difficult to iterate and update, making it unsuitable for real-time outlier detection. Finally, the method of this invention detected all outliers, demonstrating its accuracy. Further analysis revealed that both false positives occurred precisely after linear interpolation due to missing values ​​in the sequence. Furthermore, interpolation also distorted the ARIMA model to some extent, causing the final checks on the regularity and volatility of the measured values ​​to fail, resulting in false positives.

[0097] This invention proposes a method and system for real-time detection of outliers in hydrological sequences based on R-ARIMA and fluctuation tests. It comprehensively considers the short-term regularity, fluctuation rationality, and medium- to long-term iterative update requirements of hydrological monitoring data, effectively reducing the risk of misjudgment and missed judgment in outlier detection. The accuracy, robustness, and lightweight nature of the invention are demonstrated through numerical examples, and the self-iterative update of the model ensures the reliability of the invention's long-term stable operation. Thus, it provides a scientific and reasonable method for real-time outlier detection in hydrological monitoring data.

[0098] Example 2

[0099] like Figure 1 and Figure 2 As shown, the hydrological real-time sequence outlier detection system based on R-ARIMA and fluctuation test of the present invention includes a data acquisition and model building module, a difference verification module, a fluctuation verification module, and a model update module.

[0100] The data acquisition and model building module is used to acquire a preset number of hydrological measurements to form the most recent sequencing sequence, and to calculate the standard deviation of the most recent sequencing sequence; and to build an ARIMA model based on the most recent sequencing sequence.

[0101] Specifically, the data acquisition and model building module includes a data acquisition module and a model building module;

[0102] The data acquisition module is used to extract a specified number of hydrological monitoring values ​​(e.g., the most recent 72) in ascending order of time to form a recent measurement sequence, denoted as _____. And calculate the standard deviation of the sequence. ;

[0103] The model building module is used to analyze the most recent measurement sequence. Establish an ARIMA (Autoregressive Integral Moving Average) model, including a stationarity verification module, a model order determination module, and a model validation module;

[0104] The stationarity verification module is used for sequence preprocessing and stationarity testing. It employs the ADF (Augmented Dickey-Fuller) test as the algorithm for testing the stationarity of time series data, by analyzing the most recent measured sequences. Continue performing differencing operations. If the ADF test statistic is less than the preset stationarity test critical value, then the most recent measurement sequence is determined. If the sequence is stable, continue differencing until the most recent measurement sequence is reached. Until the condition stabilizes, the difference order is obtained. The stationary sequence after difference is then analyzed. Subsequently, the Ljung-Box Q test is used as the white noise test algorithm to test the stationary sequence after difference for white noise. If the test result is a non-white noise sequence, it indicates that there is effective information in the stationary sequence after difference that can be used for modeling, and a non-white noise measurement sequence is obtained to continue to the next step. If the stationary sequence after difference is white noise, the data needs to be re-examined or the sequence length needs to be adjusted.

[0105] The model order determination module is used to determine the optimal autoregressive order of the non-white noise measurement sequence based on the Bayesian Information Criterion (BIC). and optimal moving average order ;

[0106] The model validation module is used to validate models based on deterministic... The parameters are used to build an ARIMA model using maximum likelihood estimation, and the residuals of the ARIMA model are tested for white noise. If the test passes, the ARIMA model is output for prediction; otherwise, the parameters are adjusted. ;

[0107] The difference verification module is used to receive the latest measurement value. The ARIMA model is used to generate predicted values. For the predicted value Compared with the latest measurement Perform a Grubbs difference check; if the check passes, the latest measurement is determined. If the value is within the normal range, proceed to the model update module. If the test fails, then determine the latest measured value. Compared with the predicted value If significant differences are found, the system will proceed to the volatility verification module for a second test.

[0108] Specifically, the difference verification module includes a data receiving module and a statistical determination module;

[0109] The data receiving module is used to receive the latest measurement values. The ARIMA model is used to generate predicted values. ;

[0110] The statistical determination module is used to test the latest measured values ​​using the Grubbs criterion. Compared with the predicted value To determine the differences between them, construct the Grubbs difference test statistic. The formula is:

[0111] ;

[0112] in, The most recent measurement sequence Standard deviation;

[0113] At the preset significance level (Usually taken as 0.05), if the difference test statistic... Greater than or equal to the preset Grubbs threshold , The most recent measurement sequence The number of measurements, i.e. Then determine the latest measured value. Compared with the predicted value If a significant difference exists, the system proceeds to the volatility verification module for a secondary test; if the difference verification statistic... Less than Grubbs threshold ,Right now Then determine the latest measured value. If the value is normal, proceed to the model update module;

[0114] The volatility verification module is used to verify the predicted values. Compared with the latest measurement Perform a Dixon volatility test; if the test passes, the latest measurement is considered valid. If the value is within the normal range, proceed to the model update module. If the test fails, then determine the latest measured value. This is an outlier;

[0115] Specifically, if the Grubbs difference test fails, it indicates that the latest measurement may deviate from the long-term trend of the sequence. To distinguish between a genuine anomaly and a reasonable localized sharp fluctuation (such as a flood), a volatility test is required. The volatility test module includes:

[0116] The latest measured value was determined using the Dixon test criterion. Does it belong to the most recent measurement sequence? To determine the reasonable range of fluctuations, construct the Dixon volatility test statistic. The formula is:

[0117] ;

[0118] These are the latest measurements. The nth measurement in the most recent measurement sequence, For the (n-1)th measurement in the most recent measurement sequence, The most recent measurement sequence The maximum value among the differences between two consecutive terms. The most recent measurement sequence The minimum value among two consecutive differences;

[0119] At the same significance level If the volatility test statistic is below Greater than or equal to the pre-set critical value of the Dixon test (This was obtained from the Dixon discard quotient critical value table), that is... The latest measured value Volatility and recent measurement series Significant inconsistency, determine the latest measurement. This is an outlier; we await processing the next new measurement. If the volatility test statistic... Less than the critical value of the Dixon test ,Right now This indicates the latest measured value. Although it deviates from the trend, it is within a reasonable fluctuation and should be judged as a normal value, then proceed to the update model module;

[0120] The update model module is used to display the latest measurements in a sliding window. Add to recent measurement sequence In the process, the standard deviation of the most recent measurement sequence is recalculated, and the ARIMA model is updated online using the residual gradient method; then the difference verification module is returned to receive the new latest measurement and verify it.

[0121] Specifically, the model update module includes a sliding window module, a residual gradient update module, and an iteration module;

[0122] The sliding window module is used to display the latest measurement value in a sliding window manner. Add to recent measurement sequence In this approach, the oldest data point is removed, the latest measurement is added to maintain a fixed length, and the standard deviation of the most recent measurement sequence is recalculated. ;

[0123] The residual gradient update module is used to perform online iterative updates to the ARIMA model using the residual gradient method. The update formula is as follows:

[0124] ;

[0125] in, These are the updated ARIMA model coefficients. These are the coefficients of the ARIMA model before the update. For dynamic learning rate, loss function The gradient of the model coefficients, the loss function From the predicted residual Sure, The prediction residual is the difference between the latest measured value and the predicted value, i.e. ;

[0126] The dynamic learning rate is based on the sigmoid function and decays with the number of iterations.

[0127] The iteration module is used to return the difference verification module, which receives the latest measurement value and verifies it.

[0128] Since Example 2 is a modular product representation of the monitoring method in Example 1, the experiments corresponding to Example 2 are the same as those in Example 1, and will not be repeated here.

Claims

1. A method for detecting outliers in real-time hydrological sequences based on R-ARIMA and fluctuation test, characterized in that, include: S1. Obtain a preset number of hydrological measurements to form the most recent sequencing sequence, and construct an ARIMA model based on the most recent sequencing sequence. S2. Receive the latest measured value and use the ARIMA model to generate a predicted value. Perform Grubbs difference verification on the predicted value and the latest measured value. If the verification passes, the latest measured value is determined to be a normal value and proceed to S4. If the verification fails, proceed to S3 for a second verification. S3. Perform a Dixon volatility test on the predicted value and the latest measured value. If the test passes, the latest measured value is determined to be a normal value and proceed to S4. If the test fails, the latest measured value is determined to be an abnormal value. S4. Add the latest measurement to the recent measurement sequence using a sliding window, and iteratively update the ARIMA model using the residual gradient method.

2. The method for detecting outliers in real-time hydrological sequences based on R-ARIMA and fluctuation test according to claim 1, characterized in that, The construction of the ARIMA model based on the most recent sequencing sequence includes: The ADF test was used to analyze the most recent measurement sequence. Continue performing the difference operation until the most recent measured sequence is reached. Until the condition stabilizes, the difference order is obtained. The stationary sequence after difference; Perform a white noise test on the stationary sequence after difference. If the test result is a non-white noise sequence, then obtain a non-white noise measurement sequence; otherwise, re-examine the data or adjust the sequence length. Determine the optimal autoregressive order of the non-white noise measurement sequence based on the Bayesian information criterion. and optimal moving average order ; Based on certainty The parameters are set, an ARIMA model is built using maximum likelihood estimation, and the residuals of the ARIMA model are subjected to a white noise test. If the test passes, the ARIMA model is output; otherwise, the parameters are adjusted. .

3. The method for detecting outliers in real-time hydrological sequences based on R-ARIMA and fluctuation test according to claim 1, characterized in that, The Grubbs difference check includes: Constructing the Grubbs difference test statistic The formula is: ； in, The most recent measurement sequence standard deviation For predicted values, These are the latest measurements; At the preset significance level Below, if the difference test statistic Greater than or equal to the preset Grubbs threshold , The most recent measurement sequence The number of measurements, i.e. Then determine the latest measured value. Compared with the predicted value If a significant difference exists, proceed to S3 for a secondary test; if the difference test statistic... Less than Grubbs threshold ,Right now Then determine the latest measured value. If the value is normal, proceed to S4.

4. The method for detecting outliers in real-time hydrological sequences based on R-ARIMA and fluctuation test according to claim 1, characterized in that, The Dixon volatility test includes: Construct the Dixon volatility test statistic The formula is: ； in, These are the latest measurements. The nth measurement in the most recent measurement sequence, For the (n-1)th measurement in the most recent measurement sequence, The most recent measurement sequence The maximum value among the differences between two consecutive terms. The most recent measurement sequence The minimum value among two consecutive differences; At the preset significance level If the volatility test statistic is below Greater than or equal to the pre-set critical value of the Dixon test ,Right now The latest measured value Volatility and recent measurement series Significant inconsistency, determine the latest measurement. It is an outlier; if the volatility test statistic is... Less than the critical value of the Dixon test ,Right now If the latest measured value is determined to be normal, proceed to S4.

5. The method for detecting outliers in real-time hydrological sequences based on R-ARIMA and fluctuation test according to claim 1, characterized in that, The residual gradient method is used to iteratively update the ARIMA model, and the formula is as follows: ； in, These are the updated ARIMA model coefficients. These are the coefficients of the ARIMA model before the update. For dynamic learning rate, loss function The gradient of the model coefficients, the loss function The prediction residual is determined by the difference between the latest measured value and the predicted value; the dynamic learning rate is based on the sigmoid function and decays with the number of iterations.

6. A real-time hydrological series outlier detection system based on R-ARIMA and fluctuation test, characterized in that, include: The data acquisition and model building module is used to acquire a preset number of hydrological measurements to form the most recent sequencing sequence, and to build an ARIMA model based on the most recent sequencing sequence. The difference verification module is used to receive the latest measured value and generate a predicted value using the ARIMA model. It performs Grubbs difference verification on the predicted value and the latest measured value. If the verification passes, the latest measured value is determined to be a normal value and enters the model update module. If the verification fails, it enters the volatility verification module for a second verification. The volatility test module is used to perform a Dixon volatility test on the predicted value and the latest measured value. If the test passes, the latest measured value is determined to be a normal value and the module will proceed to update the model. If the test fails, the latest measured value is determined to be an outlier. The model update module is used to add the latest measurement to the recent measurement sequence in a sliding window manner, and to iteratively update the ARIMA model using the residual gradient method.

7. The hydrological real-time series outlier detection system based on R-ARIMA and fluctuation test according to claim 6, characterized in that, In the data acquisition and model building module, the construction of the ARIMA model based on the most recent sequencing sequence includes: The ADF test was used to analyze the most recent measurement sequence. Continue performing the difference operation until the most recent measured sequence is reached. Until the condition stabilizes, the difference order is obtained. The stationary sequence after difference; Perform a white noise test on the stationary sequence after difference. If the test result is a non-white noise sequence, then obtain a non-white noise measurement sequence; otherwise, re-examine the data or adjust the sequence length. Determine the optimal autoregressive order of the non-white noise measurement sequence based on the Bayesian information criterion. and optimal moving average order ; Based on certainty The parameters are set, an ARIMA model is built using maximum likelihood estimation, and the residuals of the ARIMA model are subjected to a white noise test. If the test passes, the ARIMA model is output; otherwise, the parameters are adjusted. .

8. The hydrological real-time series outlier detection system based on R-ARIMA and fluctuation test according to claim 6, characterized in that, The Grubbs difference verification module includes: Constructing the Grubbs difference test statistic The formula is: ； in, The most recent measurement sequence standard deviation For predicted values, These are the latest measurements; At the preset significance level Below, if the difference test statistic Greater than or equal to the preset Grubbs threshold , The most recent measurement sequence The number of measurements, i.e. Then determine the latest measured value. Compared with the predicted value If a significant difference exists, proceed to the volatility test module for a secondary test; if the difference test statistic... Less than Grubbs threshold ,Right now Then determine the latest measured value. If the value is normal, proceed to the model update module.

9. The hydrological real-time series outlier detection system based on R-ARIMA and fluctuation test according to claim 6, characterized in that, The volatility testing module includes the Dixon volatility test, which comprises: Construct the Dixon volatility test statistic The formula is: ； in, These are the latest measurements. The nth measurement in the most recent measurement sequence, For the (n-1)th measurement in the most recent measurement sequence, The most recent measurement sequence The maximum value among the differences between two consecutive terms. The most recent measurement sequence The minimum value among two consecutive differences; At the preset significance level If the volatility test statistic is below Greater than or equal to the pre-set critical value of the Dixon test ,Right now The latest measured value Volatility and recent measurement series Significant inconsistency, determine the latest measurement. It is an outlier; if the volatility test statistic is... Less than the critical value of the Dixon test ,Right now If the latest measured value is found to be normal, the system will proceed to the model update module.

10. The hydrological real-time series outlier detection system based on R-ARIMA and fluctuation test according to claim 6, characterized in that, In the model update module, the residual gradient method is used to iteratively update the ARIMA model, and the formula is: ； in, These are the updated ARIMA model coefficients. These are the coefficients of the ARIMA model before the update. For dynamic learning rate, loss function The gradient of the model coefficients, the loss function The prediction residual is determined by the difference between the latest measured value and the predicted value; the dynamic learning rate is based on the sigmoid function and decays with the number of iterations.

Images