Method for identifying abnormal data in drainage monitoring

Abstract

The present invention discloses a method for identifying abnormal drainage monitoring data, comprising the following steps: 1. performing classification analysis on raw detection data; 2. evaluating zero values ​​and null values ​​of flow data and liquid level data in the raw detection data; 3. performing data constancy evaluation on the flow data and liquid level data; 4. performing extreme value evaluation on the flow data and liquid level data; 5. performing threshold evaluation on the flow data and liquid level data based on historical data; 6. performing Grubbs test evaluation on the flow data and liquid level data; 7. performing a method combination evaluation based on the liquid level data and the flow data, specifically evaluating the correlation between the liquid level data and the flow data through a hydrodynamic model; and 8. storing the content of the raw detection data that is not represented as abnormal data in a cleaned detection database for subsequent application. The present invention can improve the efficiency of drainage monitoring data review and improve the standardization of monitoring data review.

Description

Technical Field

[0001] The present invention relates to the technical field of computer automatic monitoring, and in particular to a method for identifying abnormal data in drainage monitoring. Background Art

[0002] Against the backdrop of comprehensively promoting digital transformation, with the increase in the total volume of drainage pipelines and the increase in functional demands on drainage systems, the original management and scheduling based on manual experience has become increasingly difficult. There is an urgent need to empower the drainage system through intelligence and conduct systematic and comprehensive management.

[0003] Monitoring data, the most crucial sensory data in the drainage industry, plays a crucial role in model calibration, decision support, and current situation analysis for digital water management. Because drainage monitoring instruments operate in harsh environments, their sensors are easily interfered with by impurities and oil contamination, leading to anomalies in the monitoring data. If these anomalies are present, the results of analysis based on these anomalies become unreliable. Therefore, it is necessary to identify anomalies in drainage monitoring data.

[0004] At present, the method for identifying abnormal drainage monitoring data mostly uses manual experience analysis to identify abnormal data one by one, but this method has the following defects:

[0005] 1. Due to the huge amount of drainage monitoring data, a single monitoring point can generate over 100,000 data points per year (assuming one data point is recorded every five minutes). This results in engineers spending a significant amount of time and effort identifying and removing abnormal data, significantly impacting the efficiency of drainage modeling.

[0006] 2. Due to the lack of a unified method for identifying abnormal monitoring data, the final results of manual verification by different engineers are inconsistent.

[0007] 3. Some engineers have tried to use extreme values ​​to identify anomalies in monitoring data. However, since drainage monitoring data varies greatly under different working conditions (such as dry season / rainy season), simply using extreme values ​​for judgment will result in a large number of misjudgments and missed judgments.

[0008] Therefore, how to efficiently and accurately identify abnormal drainage monitoring data in batches has become a technical problem that technical personnel in this field urgently need to solve. Summary of the Invention

[0009] In view of the above-mentioned defects of the prior art, the present invention provides a method for identifying abnormal drainage monitoring data, the purpose of which is to improve the efficiency of drainage monitoring data review and the standardization of monitoring data review, thereby ultimately improving the quality of the drainage monitoring data finally obtained and providing high-quality perception data for the digital transformation of the drainage industry.

[0010] To achieve the above object, the present invention discloses a method for identifying abnormal drainage monitoring data, comprising the following steps:

[0011] Step 1: Perform classification analysis on the original test data;

[0012] Step 2: Evaluate the zero values ​​and null values ​​of the flow data and liquid level data in the original detection data to determine whether the monitored flow data and liquid level data are 0 or null, i.e., data is offline; if the monitored flow data or liquid level data is 0 or null, then identify the corresponding flow data or the corresponding liquid level data as abnormal data;

[0013] Step 3: Performing data constancy evaluation on the flow data and the liquid level data;

[0014] Step 4: Perform extreme value evaluation on the flow data and the liquid level data;

[0015] Step 5: Perform threshold evaluation on the flow data and the liquid level data based on historical data;

[0016] Step 6: Perform a Grubbs test on the flow data and the liquid level data;

[0017] Step 7: performing a method combination evaluation based on the liquid level data and the flow rate data, specifically evaluating the correlation between the liquid level data and the flow rate data through a hydrodynamic model;

[0018] Step 8: storing the content of the original detection data that is not represented as the abnormal data into the cleaned detection database for use in the next step.

[0019] Preferably, step 1 comprises the following steps:

[0020] Step 1.1, the data of each day in the original detection data are divided into four time periods, specifically: the data from 6:40-8:10 every day are divided into the morning peak period; the data from 8:15-19:00 every day are divided into the daytime period; the data from 19:05-24:00 every day are divided into the evening peak period; the data from 00:05-6:35 every day are divided into the night period;

[0021] Step 1.2, determining whether the total number of millimeters of rainfall in the data of each day in the original detection data exceeds 5 mm; if it exceeds, the next day is a rainy day; if not, the next day is a dry day;

[0022] Step 1.3: Divide the original detection data into the first to eighth categories:

[0023] Category 1: Morning peak period on dry days;

[0024] Category 2: dry daytime period;

[0025] Category 3: evening peak hours on dry days;

[0026] Category 4: dry weather night time;

[0027] Category 5: Morning rush hour on rainy days;

[0028] Category 6: Rainy daytime period;

[0029] Category 7: Evening rush hour on rainy days;

[0030] Category 8: Rainy night time.

[0031] Preferably, the data constancy evaluation in step 3 is specifically as follows: if the monitored flow data or the liquid level data maintains a constant value for more than 30 minutes, the corresponding flow data or the corresponding liquid level data are marked as abnormal data.

[0032] Preferably, the extreme value evaluation in step 4 is specifically as follows:

[0033] For the flow data, the minimum flow rate is 0, and the maximum flow rate is 1.5 times the theoretical flow rate of the corresponding section of the pipeline. Therefore, the calculation formula for the theoretical full pipe flow rate is:

[0034]

[0035] Where Q is the flow rate, in m 3 / s; D is the pipe diameter; n is the Manning coefficient; i is the pipe slope;

[0036] When the flow data in the original detection data exceeds the range from the minimum flow value to the maximum flow value, the corresponding flow data is abnormal data;

[0037] For the liquid level data, the minimum liquid level value is 0, and the maximum liquid level value is the diameter of the corresponding section of the pipeline. When the liquid level data in the original detection data exceeds the range from the minimum liquid level value to the maximum liquid level value, the data is identified as the abnormal data.

[0038] More preferably, the threshold evaluation based on historical data in step 5 is specifically as follows:

[0039] Step 5.1, extract the normal test data that has been manually reviewed in the past year corresponding to the first to the eighth categories to obtain 8 historical data;

[0040] Step 5.2, sequentially extracting the maximum historical flow value, the maximum historical liquid level value, the minimum historical flow value, and the minimum historical liquid level value corresponding to the flow data and the liquid level data in each of the historical data;

[0041] Step 5.3: Compare each of the data corresponding to the first to eighth categories in the original detection data with the corresponding maximum historical flow value and the minimum historical flow value, or the corresponding maximum historical liquid level value and the minimum historical liquid level value; any data that exceeds the range is identified as the abnormal data.

[0042] Preferably, the Grubbs test evaluation in step 6 is as follows:

[0043] Step 6.1: Establish the statistical formula for the Grubbs test evaluation, as follows:

[0044]

[0045] Among them, Y is the monitored object data; is the average value of the overall monitoring data; σ is the standard deviation of the data; G is the Grubbs coefficient;

[0046] Step 6.2: The indicator for determining whether the flow data and the liquid level data in each of the original detection data are abnormal data is: whether the calculated G value exceeds a threshold. The calculation formula of the threshold is as follows:

[0047]

[0048] Where: N is the number of monitoring data; In this case, it is a 95% confidence interval;

[0049] When G calculated from any of the flow data or the liquid level data exceeds a threshold, the corresponding flow data or the corresponding liquid level data is identified as abnormal data.

[0050] Preferably, the process of evaluating the correlation between the liquid level data and the flow data by using the hydrodynamic model in step 7 is specifically as follows:

[0051] Step 7.1. Establish a pipeline hydrodynamic mechanism model corresponding to the scope to be evaluated;

[0052] Step 7.2: using each flow data as input data of the pipeline hydrodynamic mechanism model to simulate the corresponding liquid level data;

[0053] Step 7.3: Compare each simulated liquid level data with the corresponding liquid level data in the original detection data. When the error exceeds 20%, mark the corresponding liquid level data and the corresponding flow data in the original detection data as abnormal data.

[0054] More preferably, the pipeline hydrodynamic mechanism model in step 7.1 is a SWMM model, an InfoWorks ICM model or a MIKE Urban model.

[0055] Preferably, the next step application is WEB data display, data analysis and mining and / or drainage model calibration and verification.

[0056] Preferably, the labels used to identify the abnormal data of the zero and null value evaluation, the data constancy evaluation, the extreme value evaluation, the threshold evaluation based on historical data, the Grubbs test evaluation and the hydrodynamic model evaluation are all the same.

[0057] Beneficial effects of the present invention:

[0058] The present invention can not only improve the efficiency of drainage monitoring data review, but also improve the standardization of monitoring data review, thereby ultimately improving the quality of the drainage monitoring data finally obtained.

[0059] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A flow chart showing an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0061] Example

[0062] like Figure 1 As shown, the method for identifying abnormal drainage monitoring data includes the following steps:

[0063] Step 1: Perform classification analysis on the original test data;

[0064] Step 2: Perform zero and null value evaluation on the flow data and liquid level data in the original detection data to evaluate whether the monitored flow data and liquid level data are 0 or null, that is, the data is offline; if the monitored flow data or liquid level data is 0 or null, the corresponding flow data or the corresponding liquid level data are marked as abnormal data;

[0065] Step 3: Perform data constancy evaluation on the flow data and liquid level data;

[0066] Step 4: Perform extreme value evaluation on flow data and liquid level data;

[0067] Step 5: Perform threshold evaluation on flow data and liquid level data based on historical data;

[0068] Step 6: Perform Grubbs test on the flow data and liquid level data;

[0069] Step 7: Combine the liquid level data and flow data to evaluate the hydrodynamic model and determine the correlation between the liquid level data and the flow data;

[0070] Step 8: Store the content in the original detection data that is not represented as abnormal data into the cleaned detection database for use in the next step.

[0071] The principle of the present invention is:

[0072] Drainage monitoring data, including flow and level monitoring data, is first categorized and analyzed. Six methods are then applied to identify anomalies based on the different types of data. These include zero and null value identification, data constancy, extreme value evaluation, historical data-based threshold evaluation, and the Grubbs test, which are used to verify flow and level data, respectively. Finally, a hydrodynamic model evaluation combines flow and level data for correlation verification. If any of the monitoring data is identified as abnormal by any of these methods, it is marked as such. Otherwise, normal data is identified and stored in the database. The cleaned monitoring database can be used for web-based data display, big data analysis and mining, and drainage model calibration and verification.

[0073] The present invention can quickly identify abnormal data in drainage monitoring data, thereby obtaining normal monitoring data. The larger the amount of monitoring data, the longer the processing time. According to actual project tests, the present invention can process a year's worth of monitoring data from 100 monitoring points (approximately 10 million data points) in 30 minutes.

[0074] The present invention adds a label to all monitoring data to distinguish normal data from abnormal data.

[0075] In some embodiments, step 1 includes the following steps:

[0076] Step 1.1: Divide the data of each day in the original detection data into four time periods: the data from 6:40-8:10 every day is divided into the morning peak period; the data from 8:15-19:00 every day is divided into the daytime period; the data from 19:05-24:00 every day is divided into the evening peak period; the data from 00:05-6:35 every day is divided into the night period;

[0077] Step 1.2: Determine whether the total number of millimeters of rainfall in each day of the original detection data exceeds 5 mm; if it exceeds, the next day is a rainy day; if not, the next day is a dry day;

[0078] Step 1.3: Divide the original test data into the first to eighth categories:

[0079] Category 1: Morning peak period on dry days;

[0080] Category 2: dry daytime period;

[0081] Category 3: evening peak hours on dry days;

[0082] Category 4: dry weather night time;

[0083] Category 5: Morning rush hour on rainy days;

[0084] Category 6: Rainy daytime period;

[0085] Category 7: Evening rush hour on rainy days;

[0086] Category 8: Rainy night time.

[0087] In actual applications, the drainage patterns at different times of the day are inconsistent: for example, the drainage volume during the morning rush hour of 6-8 am will be significantly higher than that during the morning of 2-4 am.

[0088] Furthermore, the drainage patterns on dry days and rainy days are different. Regardless of the drainage system, rainfall on rainy days will have a significant impact on the flow and liquid level in the drainage system. Therefore, it is necessary to classify dry days and rainy days for analysis.

[0089] In some embodiments, the data constancy evaluation in step 3 is specifically as follows: if the monitored flow data or liquid level data maintains a constant value for more than 30 minutes, the corresponding flow data or the corresponding liquid level data are marked as abnormal data.

[0090] In practical applications, for drainage monitoring data, long-term constant monitoring data indicates that there is a problem with the monitoring equipment. For example, for ultrasonic monitoring equipment, when the flow in the system becomes smaller, the measurement accuracy of some equipment will drop significantly, and then the monitored liquid level and flow will become a constant data.

[0091] In some embodiments, the extreme value evaluation in step 4 is specifically as follows:

[0092] For flow data, the minimum flow rate is 0, and the maximum flow rate is 1.5 times the theoretical flow rate of the corresponding section of the pipeline. Therefore, the calculation formula for the theoretical full pipe flow rate is:

[0093]

[0094] Where Q is the flow rate, in m 3 / s; D is the pipe diameter; n is the Manning coefficient; i is the pipe slope;

[0095] When the flow data in the original detection data exceeds the range from the minimum flow value to the maximum flow value, the corresponding flow data is abnormal data;

[0096] For liquid level data, the minimum liquid level value is 0, and the maximum liquid level value is the diameter of the corresponding section of the pipeline. When the liquid level data in the original detection data exceeds the range from the minimum liquid level value to the maximum liquid level value, it is marked as abnormal data.

[0097] In some embodiments, the threshold evaluation based on historical data in step 5 is specifically as follows:

[0098] Step 5.1: Extract the normal test data that has been manually reviewed in the past year, corresponding to the first to eighth categories, to obtain 8 historical data;

[0099] Step 5.2, extract the maximum historical flow value, maximum historical liquid level value, minimum historical flow value and minimum historical liquid level value corresponding to the flow data and liquid level data in each historical data in turn;

[0100] Step 5.3: Compare each data corresponding to the first to eighth categories in the original detection data with the corresponding maximum historical flow value and minimum historical flow value, or the corresponding maximum historical liquid level value and minimum historical liquid level value; any data outside the range is marked as abnormal data.

[0101] In practical applications, for flow and liquid level data, the threshold range can be further determined based on historical monitoring data. However, it should be noted that the data threshold ranges for different working conditions (such as dry days and rainy days) will vary greatly. Therefore, it is necessary to first classify the historical monitoring data into 8 categories according to the above monitoring data classification method.

[0102] In certain embodiments, the Grubbs test evaluation in step 6 is as follows:

[0103] Step 6.1: Establish the statistical formula for Grubbs test evaluation as follows:

[0104]

[0105] Among them, Y is the monitored object data; is the average value of the overall monitoring data; σ is the standard deviation of the data; G is the Grubbs coefficient;

[0106] Step 6.2: The indicator for determining whether the flow data and liquid level data in each raw detection data are abnormal data is: whether the calculated G value exceeds the threshold. The calculation formula of the threshold is as follows:

[0107]

[0108] Where: N is the number of monitoring data; In this case, it is a 95% confidence interval;

[0109] When G calculated from any flow data or liquid level data exceeds a threshold, the corresponding flow data or the corresponding liquid level data is identified as abnormal data.

[0110] In some embodiments, the process of evaluating the correlation between the liquid level data and the flow rate data using the hydrodynamic model in step 7 is as follows:

[0111] Step 7.1. Establish a pipeline hydrodynamic mechanism model corresponding to the scope to be evaluated;

[0112] Step 7.2: Use each flow rate data as input data of the pipeline hydrodynamic mechanism model to simulate the corresponding liquid level data;

[0113] Step 7.3: Compare each simulated liquid level data with the corresponding liquid level data in the original detection data. When the error exceeds 20%, the corresponding liquid level data and the corresponding flow data in the original detection data are marked as abnormal data.

[0114] In some embodiments, the pipeline hydrodynamic mechanism model in step 7.1 is a SWMM model, an InfoWorks ICM model, or a MIKE Urban model.

[0115] In some embodiments, the next application is WEB-side data display, data analysis and mining, and / or drainage model calibration and verification.

[0116] In certain embodiments, the labels used to identify anomalous data for zero and null value evaluations, data constancy evaluations, extreme value evaluations, historical data based threshold evaluations, Grubbs test evaluations, and hydrodynamic model evaluations are uniformly distributed.

[0117] In practical applications, abnormal data is identified by setting different tags to indicate the evaluation in which the abnormal data is generated. This makes it easier for subsequent engineers to analyze the abnormal data and find out the cause of the abnormality.

[0118] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A method for identifying abnormal drainage monitoring data; characterized in that: The steps include: Step 1: Perform classification analysis on the original test data; Step 2: Evaluate the zero values ​​and null values ​​of the flow data and liquid level data in the original detection data to determine whether the monitored flow data and liquid level data are 0 or null, i.e., data is offline; if the monitored flow data or liquid level data is 0 or null, then identify the corresponding flow data or the corresponding liquid level data as abnormal data; Step 3: Performing a data constancy assessment on the flow data and the liquid level data; the data constancy assessment is specifically as follows: if the monitored flow data or the liquid level data maintains a constant value for more than 30 minutes, the corresponding flow data or the corresponding liquid level data is marked as abnormal data; Step 4: Perform extreme value evaluation on the flow data and the liquid level data; the extreme value evaluation is specifically as follows: For the flow data, the minimum flow rate is 0, and the maximum flow rate is 1.5 times the theoretical flow rate of the corresponding section of the pipeline. Therefore, the calculation formula for the theoretical full pipe flow rate is: ； in, Q is the flow rate, in m3 / s; D is the pipe diameter; n is the Manning coefficient; i is the pipeline slope; When the flow data in the original detection data exceeds the range from the minimum flow value to the maximum flow value, the corresponding flow data is abnormal data; For the liquid level data, the minimum liquid level value is 0, and the maximum liquid level value is the diameter of the corresponding section of the pipeline. When the liquid level data in the original detection data exceeds the range from the minimum liquid level value to the maximum liquid level value, the data is marked as abnormal data; Step 5: Perform threshold evaluation on the flow data and the liquid level data based on historical data; Step 6: Perform a Grubbs test on the flow data and the liquid level data; Step 7: performing a method combination evaluation based on the liquid level data and the flow rate data, specifically evaluating the correlation between the liquid level data and the flow rate data through a hydrodynamic model; Step 8: storing the content of the original detection data that is not represented as the abnormal data into the cleaned detection database for use in the next step.

2. The method for identifying abnormal drainage monitoring data according to claim 1, characterized in that: Step 1 includes the following steps: Step 1.1, divide the data of each day in the original detection data into four time periods, specifically: divide the data from 6:40-8:10 every day into the morning peak period; divide the data from 8:15-19:00 every day into the daytime period; divide the data from 19:05-24:00 every day into the evening peak period; divide the data from 00:05-6:35 every day into the night period; Step 1.2, determining whether the total number of millimeters of rainfall in the data of each day in the original detection data exceeds 5 mm; if it exceeds, the next day is a rainy day; if not, the next day is a dry day; Step 1.3: Divide the original detection data into the first to eighth categories: Category 1: Morning peak period on dry days; Category 2: dry daytime period; Category 3: evening peak hours on dry days; Category 4: dry weather night time; Category 5: Morning rush hour on rainy days; Category 6: Rainy daytime period; Category 7: evening rush hour on rainy days; Category 8: Rainy night time.

3. The method for identifying abnormal drainage monitoring data according to claim 2, characterized in that: The threshold evaluation based on historical data described in step 5 is as follows: Step 5.1, extract the normal test data that has been manually reviewed in the past year corresponding to the first to the eighth categories to obtain 8 historical data; Step 5.2, sequentially extracting the maximum historical flow value, the maximum historical liquid level value, the minimum historical flow value, and the minimum historical liquid level value corresponding to the flow data and the liquid level data in each of the historical data; Step 5.3: Compare each of the data corresponding to the first to eighth categories in the original detection data with the corresponding maximum historical flow value and the minimum historical flow value, or the corresponding maximum historical liquid level value and the minimum historical liquid level value; any data that exceeds the range is identified as the abnormal data.

4. The method for identifying abnormal drainage monitoring data according to claim 1, characterized in that: The Grubbs test described in step 6 evaluates the following: Step 6.1: Establish the statistical formula for the Grubbs test evaluation, as follows: ； Among them, Y is the monitored object data; is the average value of the overall monitoring data; σ is the standard deviation of the data; G value is the Grubbs coefficient; Step 6.2: The indicator for determining whether the flow data and the liquid level data in each of the original detection data are abnormal data is: whether the calculated G value exceeds a threshold. The calculation formula of the threshold is as follows: ； Where: N is the number of monitoring data; ∂ in this example is the 95% confidence interval; When G calculated from any of the flow data or the liquid level data exceeds a threshold, the corresponding flow data or the corresponding liquid level data is identified as abnormal data.

5. The method for identifying abnormal drainage monitoring data according to claim 1, characterized in that: The process of evaluating the correlation between the liquid level data and the flow rate data by using the hydrodynamic model in step 7 is specifically as follows: Step 7.

1. Establish a pipeline hydrodynamic mechanism model corresponding to the scope to be evaluated; Step 7.2: using each flow data as input data of the pipeline hydrodynamic mechanism model to simulate the corresponding liquid level data; Step 7.3: Compare each simulated liquid level data with the corresponding liquid level data in the original detection data. When the error exceeds 20%, mark the corresponding liquid level data and the corresponding flow data in the original detection data as abnormal data.

6. The method for identifying abnormal drainage monitoring data according to claim 5, characterized in that: The pipeline hydrodynamic mechanism model described in step 7.1 is a SWMM model, an InfoWorks ICM model, or a MIKE Urban model.

7. The method for identifying abnormal drainage monitoring data according to claim 1, characterized in that: The next step of application is WEB data display, data analysis and mining and / or drainage model calibration and verification.

8. The method for identifying abnormal drainage monitoring data according to claim 1, characterized in that: The labels used to identify the abnormal data for the zero and null value evaluation, the data constancy evaluation, the extreme value evaluation, the historical data based threshold evaluation, the Grubbs test evaluation, and the hydrodynamic model evaluation are all the same.

Images