A method and system for cleaning and processing hydrological element data

Abstract

This invention relates to the field of hydrological monitoring technology, specifically to a method and system for cleaning and processing hydrological element data. The method includes: using a slope difference algorithm to perform initial filtering on hydrological data to obtain preliminary cleaned data; obtaining fluctuation parameters of the preliminary cleaned data; determining whether the preliminary cleaned data is fluctuating based on the fluctuation parameters; if the preliminary cleaned data is not fluctuating, using a first mean filtering algorithm to clean the preliminary cleaned data to obtain final cleaned data; if the preliminary cleaned data is fluctuating, detecting whether the fluctuation data is normal fluctuation data; if the fluctuation data is normal fluctuation data, using a second mean filtering algorithm to clean the preliminary cleaned data to obtain final cleaned data; if the fluctuation data is abnormal fluctuation data, determining the preliminary cleaned data as the final cleaned data. This disclosure comprehensively improves the accuracy of data and greatly reduces the impact of erroneous data on the accuracy of analysis results.

Description

Technical Field

[0001] This invention relates to the field of hydrological monitoring technology, specifically to a method and system for cleaning and processing hydrological element data. Background Technology

[0002] In the field of hydrological monitoring, data quality is crucial for decision-making in flood and drought disaster prevention, water resource management, and environmental protection. With the technological advancements in the water industry and the increasing demands for water resource management, the accuracy and reliability of hydrological data, as a vital basis for water resource management, have become particularly critical. This data is used not only for routine water resource allocation and flood and drought early warning but also extensively in long-term water resource planning and environmental impact assessments.

[0003] Due to equipment failure, environmental interference, network transmission issues, etc., the data recorded by hydrological stations may experience abnormal fluctuations or sudden changes.

[0004] However, these abnormal fluctuations or sudden changes in hydrological data not only affect the accuracy of data analysis, but also mislead decision-makers' judgments, leading to inappropriate problem-solving measures.

[0005] Due to the unique properties of hydrological data, traditional data cleaning methods may not be able to effectively identify and handle outliers in such data, such as unrealistic data fluctuations or abrupt changes, which may lead to erroneous data affecting the accuracy of analysis results. Summary of the Invention

[0006] To address the technical problem of abnormal fluctuations or sudden changes in hydrological data affecting the accuracy of data analysis, the present invention aims to provide a method and system for cleaning and processing hydrological element data. The specific technical solution adopted is as follows:

[0007] In a first aspect, this disclosure provides a method for cleaning and processing hydrological element data, the method comprising:

[0008] The slope difference algorithm is used to perform initial filtering of hydrological data to obtain preliminary cleaned data.

[0009] Obtain the fluctuation parameters of the preliminary cleaning data;

[0010] Determine whether the preliminary cleaning data is fluctuating based on the fluctuation parameters;

[0011] When the preliminary cleaned data is not the fluctuating data, the first mean filtering algorithm is used to clean the preliminary cleaned data to obtain the final cleaned data;

[0012] When the initial cleaning data is the fluctuation data, it is detected whether the fluctuation data is normal fluctuation data;

[0013] If the fluctuation data is normal fluctuation data, the second mean filtering algorithm is used to clean the preliminary cleaned data to obtain the final cleaned data.

[0014] If the fluctuation data is abnormal fluctuation data, the preliminary cleaned data is determined to be the final cleaned data.

[0015] In one embodiment, after determining the final cleaned data, the method further includes:

[0016] Store the final cleaning data.

[0017] In one embodiment, the initial filtering of hydrological data using a slope difference algorithm to obtain preliminary cleaned data includes:

[0018] Obtain the change in two temporally adjacent hydrological data points from the hydrological data;

[0019] The time interval for obtaining two adjacent hydrological data points;

[0020] The slope of the two adjacent hydrological data points is obtained based on the change in the two adjacent hydrological data points and the corresponding time interval.

[0021] The target slope is obtained when the slope of two adjacent hydrological data points is greater than a preset slope threshold. The hydrological data points corresponding to the target slope are preliminary anomalies.

[0022] Obtain the water level difference between two adjacent preliminary anomaly points;

[0023] Identify target anomalies where the water level difference is greater than a preset change threshold, where the target anomalies are real anomaly data.

[0024] The preliminary cleaning data is obtained by filtering the real abnormal data from the hydrological data.

[0025] In one embodiment, obtaining the fluctuation parameters of the preliminary cleaning data includes:

[0026] Obtain the hydrological data difference between any two time-series adjacent hydrological data in the preliminary cleaning data;

[0027] The fluctuation parameters of the preliminary cleaning data are obtained based on the number of all the preliminary cleaning data and the sum of the differences of all the hydrological data.

[0028] In one embodiment, determining whether the preliminary cleaning data is fluctuating data based on the fluctuation parameter includes:

[0029] Detect whether the fluctuation parameter is greater than a preset fluctuation threshold;

[0030] If the fluctuation parameter is greater than the preset fluctuation threshold, then the preliminary cleaning data is determined to be the fluctuation data;

[0031] If the fluctuation parameter is not greater than the preset fluctuation threshold, then the preliminary cleaning data is determined not to be the fluctuation data.

[0032] In one embodiment, detecting whether the fluctuation data is normal fluctuation data includes:

[0033] Obtain the peak value of all peaks in the fluctuation data;

[0034] The sharpness of the corresponding wave peak is obtained based on the peak value.

[0035] The fluctuation data corresponding to the wave peak whose sharpness is greater than a preset sharpness threshold is identified as the abnormal fluctuation data.

[0036] The fluctuation data corresponding to the wave peak whose sharpness is not greater than the preset sharpness threshold is determined to be the normal fluctuation data.

[0037] In one embodiment, obtaining the sharpness corresponding to the peak value includes:

[0038] For each of the stated peaks, perform the following steps:

[0039] Get a preset number of other peaks adjacent to the current peak;

[0040] Obtain the center of the current peak and the other peaks;

[0041] The sharpness of the current peak is obtained based on the peak value at the center of the peak and the peak value of the current peak.

[0042] In one embodiment, the step of cleaning the preliminary cleaned data using a first mean filtering algorithm includes:

[0043] Identify any abnormal data in the initial cleaned data;

[0044] The abnormal data is filtered out from the preliminary cleaning data to obtain the final cleaning data;

[0045] The process of identifying anomalous data in the preliminary cleaned data includes:

[0046] For each piece of cleaned data in the initial cleaned data, perform the following steps:

[0047] Get the time when the current cleaning data occurred;

[0048] Acquire a preset number of historical hydrological data, filtered by the slope difference algorithm, that are the same as the occurrence time.

[0049] Obtain the mean of the historical hydrological data;

[0050] Based on the mean and the current cleaning data, obtain the mean feature difference corresponding to the current cleaning data;

[0051] If the difference in the mean features is greater than a preset mean threshold, then the current cleaned data is determined to be abnormal data.

[0052] In one embodiment, the preset mean threshold corresponding to the first mean filtering algorithm is less than the preset mean threshold corresponding to the second mean filtering algorithm.

[0053] A second aspect of this disclosure provides a hydrological element data cleaning and processing system, the system comprising:

[0054] The data acquisition module is used to collect hydrological data, including water level data.

[0055] The initial filtering module is used to perform initial filtering on the hydrological data using a slope difference algorithm to obtain preliminary cleaning data.

[0056] A secondary filtering module is used to acquire fluctuation parameters of the preliminary cleaned data; determine whether the preliminary cleaned data is fluctuating data based on the fluctuation parameters; if the preliminary cleaned data is not fluctuating data, use a first mean filtering algorithm to clean the preliminary cleaned data to obtain final cleaned data; if the preliminary cleaned data is fluctuating data, detect whether the fluctuating data is normal fluctuating data; if the fluctuating data is normal fluctuating data, use a second mean filtering algorithm to clean the preliminary cleaned data to obtain the final cleaned data; if the fluctuating data is abnormal fluctuating data, determine that the preliminary cleaned data is the final cleaned data.

[0057] A storage module is used to store the final cleaning data.

[0058] In one embodiment, the initial filtering module is specifically used for:

[0059] Obtain the change in two temporally adjacent hydrological data points from the hydrological data;

[0060] The time interval for obtaining two adjacent hydrological data points;

[0061] The slope of the two adjacent hydrological data points is obtained based on the change in the two adjacent hydrological data points and the corresponding time interval.

[0062] The target slope is obtained when the slope of two adjacent hydrological data points is greater than a preset slope threshold. The hydrological data points corresponding to the target slope are preliminary anomalies.

[0063] Obtain the water level difference between two adjacent preliminary anomaly points;

[0064] The target anomaly point is obtained when the water level difference is greater than a preset change threshold. The target anomaly point is the real anomaly data.

[0065] In one embodiment, the secondary filtering module is specifically used for:

[0066] Obtain the hydrological data difference between any two time-series adjacent hydrological data in the preliminary cleaning data;

[0067] The fluctuation parameters of the preliminary cleaning data are obtained based on the number of all the preliminary cleaning data and the sum of the differences of all the hydrological data.

[0068] In one embodiment, the secondary filtering module is specifically used for:

[0069] Detect whether the fluctuation parameter is greater than a preset fluctuation threshold;

[0070] If the fluctuation parameter is greater than the preset fluctuation threshold, then the preliminary cleaning data is determined to be the fluctuation data;

[0071] If the fluctuation parameter is not greater than the preset fluctuation threshold, then the preliminary cleaning data is determined not to be the fluctuation data.

[0072] In one embodiment, the secondary filtering module is specifically used for:

[0073] Obtain the peak value of all peaks in the fluctuation data;

[0074] The sharpness of the corresponding wave peak is obtained based on the peak value.

[0075] The fluctuation data corresponding to the wave peak whose sharpness is greater than a preset sharpness threshold is identified as the abnormal fluctuation data.

[0076] The fluctuation data corresponding to the wave peak whose sharpness is not greater than the preset sharpness threshold is determined to be the normal fluctuation data.

[0077] In one embodiment, the secondary filtering module is specifically used for:

[0078] For each of the stated peaks, perform the following steps:

[0079] Get a preset number of other peaks adjacent to the current peak;

[0080] Obtain the center of the current peak and the other peaks;

[0081] The sharpness of the current peak is obtained based on the peak value at the center of the peak and the peak value of the current peak.

[0082] In one embodiment, the secondary filtering module is specifically used for:

[0083] Identify any abnormal data in the initial cleaned data;

[0084] The abnormal data is filtered out from the preliminary cleaning data to obtain the final cleaning data;

[0085] The process of identifying anomalous data in the preliminary cleaned data includes:

[0086] For each piece of cleaned data in the initial cleaned data, perform the following steps:

[0087] Get the time when the current cleaning data occurred;

[0088] Acquire a preset number of historical hydrological data, filtered by the slope difference algorithm, that are the same as the occurrence time.

[0089] Obtain the mean of the historical hydrological data;

[0090] Based on the mean and the current cleaning data, obtain the mean feature difference corresponding to the current cleaning data;

[0091] If the difference in the mean features is greater than a preset mean threshold, then the current cleaned data is determined to be abnormal data.

[0092] In one embodiment, the preset mean threshold corresponding to the first mean filtering algorithm is less than the preset mean threshold corresponding to the second mean filtering algorithm.

[0093] The present invention has the following beneficial effects:

[0094] This disclosure, taking into account the characteristics of hydrological data, first uses a slope difference algorithm for initial filtering. Based on the slope difference of the water level data before and after the change, it can keenly identify data fluctuations and abrupt changes that deviate from the normal change trajectory, accurately identify abnormal data, and initially remove them from the original data, curbing the interference of erroneous data at the source. Then, a mean filtering algorithm is used for secondary fine cleaning. The mean filtering algorithm uses the mean of the water level data as a benchmark to comprehensively calibrate the water level data after the initial cleaning. It can smooth out minor data noise and residual deviations, comprehensively improve the accuracy of the data, and greatly reduce the impact of erroneous data on the accuracy of the analysis results. This makes the analytical conclusions drawn from the cleaned data more reliable, providing a solid data foundation for hydrological research and water conservancy engineering decision-making. Attached Figure Description

[0095] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0096] Figure 1 A flowchart illustrating a hydrological element data cleaning and processing method according to an embodiment of the present invention. Figure 1 ;

[0097] Figure 2 A flowchart illustrating a hydrological element data cleaning and processing method according to an embodiment of the present invention. Figure 2 ;

[0098] Figure 3 Slope illustration provided for one embodiment of the present invention Figure 1 ;

[0099] Figure 4 Slope illustration provided for one embodiment of the present invention Figure 2 ;

[0100] Figure 5 A flowchart illustrating a hydrological element data cleaning and processing method according to an embodiment of the present invention. Figure 3 ;

[0101] Figure 6 This is a schematic diagram of fluctuation parameters and water level fluctuations provided in one embodiment of the present invention;

[0102] Figure 7 This is a schematic diagram of a peak in the curve corresponding to the fluctuation data provided in one embodiment of the present invention;

[0103] Figure 8 This is a schematic diagram of a hydrological data cleaning and processing system provided in one embodiment of the present invention. Detailed Implementation

[0104] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a hydrological element data cleaning and processing method and system proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0105] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0106] Abnormal fluctuations or sudden changes in hydrological data not only affect the accuracy of data analysis, but can also mislead decision-makers' judgments, leading to inappropriate problem-solving measures.

[0107] Most existing data cleaning tools and methods fail to fully consider the unique attributes of hydrological data, such as river levels and flows, and reservoir levels and capacities. These parameters possess specific time-series characteristics and physical laws, and can form certain historical features when combined with historical data. Traditional data cleaning methods may be unable to effectively identify and handle outliers in such data, such as unrealistic data fluctuations or abrupt changes. This can lead to erroneous data affecting the accuracy of analysis results and also cause bias in stored historical data.

[0108] To address the aforementioned technical problems, this disclosure proposes a method and system for cleaning and processing hydrological element data.

[0109] The following description, in conjunction with the accompanying drawings, details the specific scheme of the hydrological element data cleaning and processing method and system provided by the present invention.

[0110] Please see Figure 1 The diagram illustrates a flowchart of a hydrological element data cleaning and processing method according to an embodiment of the present invention, which includes the following steps S101-S104:

[0111] Step S101: Collect hydrological data.

[0112] This step is used to obtain relevant hydrological data recorded by hydrological monitoring stations. For example, water level data.

[0113] Hydrological data for reservoirs, rivers, and other water bodies need to be obtained from monitoring stations. Therefore, the city's Water Resources Bureau's smart water management system is used to obtain raw data from hydrological stations throughout the city, including data on rainfall, surface water, groundwater, and the water environment.

[0114] In this example, we will use water level data monitored by a river station in a certain city as an example to describe the subsequent data cleaning and processing methods.

[0115] Since the river stations continuously measure and collect water level data, the above-mentioned river water level data is a stream data, so the subsequent data cleaning and processing methods are all based on the above-mentioned river water level time series data.

[0116] Due to the demands of water resource management, high-quality hydrological data is needed for water resource allocation, flood and drought early warning, and long-term water resource planning and scheduling. Therefore, before using this data, it is necessary to first clean the collected data, removing erroneous data such as large fluctuations and abrupt changes that do not conform to reality, and retaining accurate data to improve the accuracy of subsequent data analysis.

[0117] Regarding data cleaning, there are many mature technologies available. However, most existing data cleaning tools and methods do not take into account the unique properties of hydrological data, such as parameters like river water level and flow rate. These data have specific temporal characteristics and interrelated physical laws, and can be combined with historical data to form certain historical features.

[0118] Therefore, the hydrological data cleaning and processing method proposed in this invention will be described in detail below.

[0119] Step S102: Use the slope difference algorithm to perform initial filtering on the hydrological data to obtain preliminary cleaned data.

[0120] Specifically, such as Figure 2 As shown, step S102 includes the following sub-steps S1021-S1027:

[0121] S1021. Obtain the change in two adjacent hydrological data points in time series.

[0122] S1022, the time interval for obtaining two adjacent hydrological data.

[0123] S1023. Based on the change in two adjacent hydrological data and the corresponding time interval, obtain the slope of two adjacent hydrological data.

[0124] S1024. Obtain the target slope where the slope of two adjacent hydrological data points is greater than the preset slope threshold. The hydrological data corresponding to the target slope is the initial anomaly point.

[0125] S1025. Obtain the water level difference between two adjacent preliminary anomaly points.

[0126] S1026. Obtain target anomaly points where the water level difference is greater than a preset change threshold. These target anomaly points are real anomaly data.

[0127] S1027. Filter out real abnormal data from hydrological data to obtain preliminary cleaned data.

[0128] Specifically, taking water level data from river monitoring stations as an example, there is a strong correlation between two adjacent water level data points in time series, and their changing patterns can well reflect the characteristics of the water level data within that time period. Therefore, based on two adjacent water level data points in time series and time, the slope of the two adjacent water level data points is calculated:

[0129]

[0130] In the formula, The slope of two adjacent water level data points. This refers to the change in water level data between two adjacent data points in a time series. This represents the time interval between two adjacent water level data points in the time series.

[0131] Calculated slope This can represent the changing trend of these two adjacent water level data.

[0132] Under normal circumstances, the river level will not change significantly in a short period of time, therefore It should also be at a relatively low level. Therefore, a preset slope threshold can be set in this disclosure. ,for If two adjacent water level data points are identified, they are considered to be abnormal water level data points and are determined to be preliminary anomaly points.

[0133] The above method determines whether data is anomaly by calculating the slope between adjacent points in the water level-time time series. This approach typically identifies hydrological data with a large slope as anomaly. However, in reality, there are situations where data loss exists between two adjacent hydrological data points in the time series. In such cases, the water level change between the two data points might be significant, but the calculated slope may not reflect this. The difference might not be very high; if the time interval between two collected data points is very small, the difference in water level data may be small, but the calculated slope will be much higher. But it will be very large. For example... Figure 3 and Figure 4 As shown, Figure 3 and Figure 4 The horizontal axis T represents time, and the vertical axis WL represents water level.

[0134] It is necessary to combine the water level difference. and slope The data to be processed is filtered together. The specific method is as follows:

[0135] First, for a set of water level data to be processed, the slope between each point and its adjacent water level data is calculated using the method described above. ,Will Water level data exceeding the preset slope threshold are recorded as preliminary anomalies.

[0136] Next, taking the preliminary anomaly points filtered in S1024 as the target, we will further calculate their water level differences. water level difference The larger the initial anomaly, the greater the likelihood that it is a genuine anomaly; while the water level difference... The smaller the value, the more likely the initial outlier is to be normal data that has been mistakenly identified as abnormal data.

[0137] water level difference Set a preset threshold for change For the preliminary anomaly points calculated in the previous step, if their water level difference... If so, then this initial anomaly is considered genuine abnormal data and is filtered out. Regarding the preset change threshold... The threshold value needs to be selected by the experimenters based on their experience and the actual conditions of the local river. The smaller the preset threshold value, the more sensitive the algorithm is to changes in the river water level.

[0138] This completes the slope difference filtering of hydrological data. The algorithm prioritizes the data slope and uses the difference as a secondary factor, thus ensuring the identification of abnormal data while avoiding the incorrect processing of correct data.

[0139] In step S102, the currently collected hydrological data is directly analyzed and calculated to identify and clean abnormal data. Next, in this step, the collected hydrological data is used as the object of analysis, combined with the preliminary cleaned data obtained after slope difference cleaning, to further analyze the hydrological data, identify and remove abnormal data, and complete the cleaning of the hydrological data.

[0140] Step S103: Use the mean filtering algorithm to filter the preliminary cleaned data again to obtain the final cleaned data;

[0141] In step S102, the currently collected hydrological data is directly analyzed and calculated to identify and clean abnormal data. Next, in this step, the collected hydrological data is used as the object of analysis, combined with the preliminary cleaned data after slope difference cleaning, to further analyze the hydrological data, identify and remove abnormal data, and complete the hydrological data cleaning process.

[0142] Specifically, step S103 can be implemented in the following way:

[0143] A1. Identify abnormal data in the initial cleaned data.

[0144] A2. Filter out abnormal data from the initial cleaned data to obtain the final cleaned data.

[0145] The process of identifying anomalous data in the preliminary cleaned data includes performing the following steps A11-A15 on each piece of cleaned data in the preliminary cleaned data:

[0146] A11. Obtain the time when the current cleaning data occurred.

[0147] A12. Obtain a preset number of historical water level data filtered by the slope difference algorithm, which is the same as the occurrence time.

[0148] A13. Obtain the average of historical water level data.

[0149] A14. Based on the mean and the current cleaned data, obtain the mean feature difference corresponding to the current cleaned data.

[0150] A15. If the difference in mean features is greater than the preset mean threshold, then the current cleaned data is determined to be abnormal data.

[0151] Taking the water level data of river stations as an example again, the data processed by the algorithm in the previous step contains a certain amount of correct data that can be referenced. The average value of the data calculated using a certain amount of historical correct data from the same period can reflect the characteristics of the river water level at that station to a certain extent. Therefore, it can be assumed that if the difference between the average value of the next collected real-time flow cytometry water level data and the currently calculated water level data is very small, then the next water level data is considered to be correct; if the difference between the average value of the next collected real-time flow cytometry water level data and the currently calculated water level data is very large, then the water level data is considered to be incorrect.

[0152] Based on the above analysis, the specific calculation formula for mean filtering is as follows:

[0153] ;

[0154] in, Indicates the difference in mean characteristics. This represents the mean of the water level data stream after slope difference cleaning during the same period; This indicates the water level data collected in real time at the current moment.

[0155] This formula uses the absolute value of the difference between the mean of water level data after slope difference washing during the same period and the real-time water level data collected at the current moment to represent the mean characteristic gap, which measures the difference between the currently calculated water level data and the mean of water level data during the same period. The larger this value, the more likely the calculated water level data is to be abnormal.

[0156] The same applies here to the mean characteristic difference. A preset average threshold is set, and water level data exceeding the preset average threshold is considered abnormal data. Then, the abnormal data obtained by the average filtering algorithm is filtered out again from the initial cleaning data to obtain the final cleaning data.

[0157] Step S104: Store the final cleaning data.

[0158] After completing the data cleaning process described above, the next step is to store the cleaned and valid data into the database in a high-concurrency manner. Each data entry is labeled with its data status and stored in the corresponding database table according to its monitoring station, data type (such as river level, flow rate, reservoir level and capacity), and other relevant attributes.

[0159] For data information identified as abnormal during the cleaning process, it is extracted, marked, and stored in an abnormal data table. The abnormal data table records the time, type, and corresponding hydrological elements of the data, and also needs to record the reason why the current data is identified as abnormal. This is important for subsequent analysis of the cause of the abnormality, troubleshooting, and algorithm improvement. At the same time, it can serve as a historical reference to help identify abnormal situations more accurately in the future.

[0160] This disclosure, taking into account the characteristics of hydrological data, first uses a slope difference algorithm for initial filtering. Based on the slope difference of the water level data before and after the change, it can keenly identify data fluctuations and abrupt changes that deviate from the normal change trajectory, accurately identify abnormal data, and initially remove them from the original data, curbing the interference of erroneous data at the source. Then, a mean filtering algorithm is used for secondary fine cleaning. The mean filtering algorithm uses the mean of the water level data as a benchmark to comprehensively calibrate the water level data after the initial cleaning. It can smooth out minor data noise and residual deviations, comprehensively improve the accuracy of the data, and greatly reduce the impact of erroneous data on the accuracy of the analysis results. This makes the analytical conclusions drawn from the cleaned data more reliable, providing a solid data foundation for hydrological research and water conservancy engineering decision-making.

[0161] In steps S102 and S103 above, two methods for cleaning hydrological data were proposed: slope difference filtering and mean filtering. The input data used for mean filtering was obtained from slope difference filtering. It can be observed that the cleaning effect of both algorithms on hydrological data is determined by their respective thresholds. The core idea of ​​the mean filtering algorithm is that stable water level data (data closer to the mean) is normal data, while data with large fluctuations (data deviating significantly from the mean) is likely abnormal data. However, taking river water level data as an example, in reality, it may exhibit large recent fluctuations (the fluctuations referred to here are normal data fluctuations, such as reasonable rises and falls in water levels due to heavy rain, not abnormal data fluctuations). In this case, using the average water level data of the same period to measure data stability is not realistic for mean filtering. Therefore, when combining the two algorithms, it is first necessary to analyze the development trend of the input hydrological data, and then dynamically enhance or weaken the filtering effect of mean filtering to clean abnormal data while ensuring that normal data is not erroneously processed to the greatest extent possible.

[0162] Therefore, as Figure 5 As shown, step S103 above uses a mean filtering algorithm to filter the preliminary cleaned data again to obtain the final cleaned data, including the following sub-steps S1031-S1036:

[0163] S1031. Obtain the fluctuation parameters of the preliminary cleaning data.

[0164] In one embodiment, obtaining the fluctuation parameters of the preliminary cleaning data includes the following sub-steps S10311-S10312:

[0165] S10311. Obtain the hydrological data difference between any two adjacent hydrological data in the time series in the preliminary cleaning data;

[0166] S10312. Obtain the fluctuation parameters of the preliminary cleaning data based on the fluctuation base, the number of all preliminary cleaning data, and the sum of the differences of all hydrological data.

[0167] First, it is necessary to assess the volatility of the input hydrological data. During this assessment, for example... Figure 6 As shown, if the hydrological data fluctuates significantly, the fluctuation parameter will be large; if the hydrological data is relatively stable, the fluctuation parameter will be low.

[0168] Here, we also take river water level data as an example. The calculation method for the fluctuation parameters of the water level data in the time series is as follows:

[0169]

[0170] in, This parameter represents the fluctuation of the input water level data, which can measure the overall volatility of the data. This represents the difference in water level data between two adjacent water level data points in time series. The fluctuation base represents the fluctuation noise of the water level data when the water level data is in a normal state most of the time, that is, when the river water situation is stable. The amount of initial cleaned data input.

[0171] S1032. Determine whether the preliminary cleaned data is fluctuating data based on the fluctuation parameters. If the preliminary cleaned data is not fluctuating data, proceed to step S1033. If the preliminary cleaned data is fluctuating data, proceed to step S1034.

[0172] Specifically, if the fluctuation parameter is greater than the preset fluctuation threshold, the preliminary cleaned data is determined to be fluctuating data; if the fluctuation parameter is not greater than the preset fluctuation threshold, the preliminary cleaned data is determined not to be fluctuating data.

[0173] Even during periods of stable water level performance, fluctuations still occur. Therefore, we can obtain a baseline of water level fluctuations during periods of stable performance. In other words, the fluctuation range of water level data is less than At this point, the water level data is considered to be "stable". Then, the difference between two adjacent water level data points is determined. Fluctuation noise of data during the aforementioned stable period The difference between them, due to Since it is a very small value, the larger the difference, the less similar the water level data at this moment is to the past stable periods, which means that the water level data fluctuates more at this time.

[0174] The preset fluctuation threshold here can be 1, for If two adjacent water level data points are considered to be fluctuating data rather than stable data, then they are considered to be fluctuating data.

[0175] S1033. When the preliminary cleaned data is not fluctuating, the first mean filtering algorithm is used to clean the preliminary cleaned data to obtain the final cleaned data.

[0176] In one implementation, step S1033 uses a first mean filtering algorithm to clean the preliminary cleaned data, including the following sub-steps B1-B2:

[0177] B1. Identify abnormal data in the initial cleaned data.

[0178] B2. Filter out abnormal data from the initial cleaned data to obtain the final cleaned data.

[0179] The process of identifying anomalous data in the preliminary cleaned data includes performing the following steps B11-B15 on each piece of cleaned data in the preliminary cleaned data:

[0180] B11. Obtain the time when the current cleaning data occurred.

[0181] B12. Obtain a preset number of historical hydrological data filtered by the slope difference algorithm, which is the same as the occurrence time.

[0182] B13. Obtain the mean of historical hydrological data.

[0183] B14. Based on the mean and the current cleaned data, obtain the mean feature difference corresponding to the current cleaned data.

[0184] B15. If the difference in mean features is greater than the preset mean threshold, then the current cleaned data is determined to be abnormal data.

[0185] S1034. When the initial cleaned data is fluctuating data, check whether the fluctuating data is normal fluctuating data. If the fluctuating data is normal fluctuating data, proceed to step S1035. If the fluctuating data is not normal fluctuating data, proceed to step S1036.

[0186] S1035. If the fluctuation data is normal fluctuation data, the second mean filtering algorithm is used to clean the preliminary cleaned data to obtain the final cleaned data.

[0187] The implementation of the second mean filtering algorithm is similar to the mean filtering algorithm in step S103 of the above embodiment, and will not be described again here. It is worth noting that the preset mean threshold of the first mean filtering algorithm is less than the preset mean threshold of the second mean filtering algorithm.

[0188] S1036. If the fluctuation data is abnormal fluctuation data, determine the preliminary cleaned data as the final cleaned data.

[0189] Step S1031 yields fluctuation parameters that can measure the fluctuation state of the current water level data. Further analysis will be conducted using the fluctuation data as the object. This fluctuation parameter only assesses the macroscopic volatility of the data; therefore, it is also necessary to determine whether this set of fluctuation data represents normal or abnormal fluctuations.

[0190] Taking water level data collected from a fixed monitoring point on a river as an example, the river width at that point is fixed, meaning the maximum flow rate at that point is determined. Therefore, if precipitation occurs, the fluctuation curve of the water level at that point from rising to falling should be similar to the fluctuation of water level data after precipitation in historical data. Furthermore, assuming two sets of water level data after precipitation, if the river flow did not exceed the maximum flow rate after either precipitation, then the curves of these two sets of water level data should show only differences in the amplitude of the fluctuations, while the trends and shapes are similar. Moreover, according to common sense, the rise and fall of river water level after precipitation is a clearly uniform trend. However, for abnormal data caused by acquisition errors, sensor malfunctions, etc., it will appear as abrupt discrete points on the entire data set, and it is almost impossible to find similar contemporaneous data in historical data.

[0191] In one embodiment, detecting whether the fluctuation data is normal fluctuation data in step S1034 includes the following sub-steps S10341-S10344:

[0192] S10341. Obtain the peak value of all peaks in the fluctuation data.

[0193] S10342. Obtain the sharpness of the corresponding peak based on the peak value.

[0194] Specifically, the sharpness of the peak is obtained based on the peak value, including performing the following sub-steps C1-C3 for each peak:

[0195] C1. Obtain the peak value of a preset number of other peaks adjacent to the current peak.

[0196] C2. Obtain the peak center between the current peak and other peaks.

[0197] C3. Obtain the sharpness of the peak value of the current peak based on the peak value at the center of the peak and the peak value of the current peak.

[0198] After obtaining the peak values ​​of all peaks in the fluctuation data, taking one peak in the currently collected fluctuation data as an example, the sharper the peak, the more likely the peak value is to be an anomaly.

[0199] Therefore, the method for calculating the sharpness of each peak in the fluctuation data is as follows:

[0200]

[0201] in, This indicates the sharpness of the peak values ​​of each wave in the fluctuation data. This represents the current peak value in the curve corresponding to the fluctuation data. For all peaks in the curve corresponding to the fluctuation data, This indicates the peak immediately preceding the current peak. Indicates peak The peak value. For a given peak value , This represents all discrete regions encompassed by the current peak value, where the peaks within these discrete regions are a preset number of other peaks adjacent to the current peak value. It is the location of the center of the wave crest. For example... Figure 7 As shown, it illustrates a peak in the curve corresponding to the fluctuation data.

[0202] The greater the difference between the values ​​on both sides of a peak and the peak value, the narrower the peak is, meaning the peak value is sharper. This means for the peak value Calculate its adjacent two sides All With the center of the peak The sum of the squares of the differences, multiplied by That is, all peaks in the curve corresponding to the abnormal data. .

[0203] S10343. Determine the fluctuation data corresponding to the peak with a sharpness greater than the preset sharpness threshold as abnormal fluctuation data.

[0204] S10344. Determine the fluctuation data corresponding to the peak whose sharpness is not greater than the preset sharpness threshold as normal fluctuation data.

[0205] The larger the value, the greater the difference between the values ​​on both sides of the peak of the curve corresponding to the fluctuation data and the peak, the steeper the peak, and the more likely the water level peak data is to be abnormal data.

[0206] In this way, the anomalies of several extreme points in the collected fluctuation data are obtained, and a preset sharpness threshold is set. ,for If the peak value is not found, it is considered to be an abnormal fluctuation rather than a normal fluctuation.

[0207] The calculation method for the sharpness of the trough is exactly the same as that for the peak. Specifically, step S1034 checks whether the fluctuation data is normal fluctuation data, including the following sub-steps D1-D4:

[0208] D1. Obtain the valley value of all valleys in the fluctuation data.

[0209] D2. Obtain the sharpness of the corresponding trough based on the trough value.

[0210] Specifically, based on the valley value, the sharpness of the corresponding valley is obtained, including performing the following sub-steps D11-D13 for each valley:

[0211] D11. Obtain the valley values ​​of a preset number of other valleys adjacent to the current valley.

[0212] D12. Obtain the center of the current trough and other troughs.

[0213] D13. Obtain the sharpness of the current trough value based on the trough value at the center of the trough and the current trough value.

[0214] D3. Determine the fluctuation data corresponding to the troughs whose sharpness exceeds the preset sharpness threshold as abnormal fluctuation data.

[0215] D4. Determine the fluctuation data corresponding to the troughs whose sharpness is not greater than the preset sharpness threshold as normal fluctuation data.

[0216] As we know from the previous analysis, when data fluctuations are large, i.e., when the fluctuation parameter is large, the water level data may fall into two categories: reasonable rises and falls in water level due to natural phenomena, or abnormal data fluctuations caused by other reasons. We also calculate the sharpness of the extreme values ​​in the fluctuation data curve. The two types of fluctuating data mentioned above have been distinguished. However, during subsequent data cleaning, the feature mean calculated by mean filtering cannot accurately clean the fluctuating data. Specifically, for normal and reasonable data fluctuations, we only need to process extremely rare values; for abnormal fluctuations, the outliers themselves have a significant impact on the feature mean, so the calculated feature mean cannot accurately represent the correct data. Therefore, the effect (weight) of mean filtering needs to be reduced. When the fluctuations are very large, the feature mean calculated based on the water level of the same period becomes meaningless, so mean filtering should be discontinued, and slope difference filtering should only be used on the original water level data.

[0217] Based on the above analysis, the hydrological data cleaning method after combining the two algorithms is as follows:

[0218] First, the collected hydrological data is cleaned by using slope difference filtering.

[0219] Next, the fluctuation parameters of the data processed by the slope difference filtering algorithm are calculated. First, determine if the data is fluctuating. If not, use mean filtering to further clean the data. If it is, determine whether it is normal or abnormal fluctuation data. For normal fluctuation data, increase the threshold of the feature mean to filter only those extremely rare values ​​that deviate significantly from the majority of the data. For abnormal fluctuation data, simply set the threshold of the feature mean to a large number, meaning that mean filtering is not used in the processing.

[0220] This completes the combination of slope difference filtering and mean filtering described above: first, slope difference filtering is applied to the collected data, and then the fluctuation parameters of the cleaned data are calculated. And based on the fluctuation parameters By identifying the sharpness of peaks in fluctuating data to determine the data type, the threshold of the feature mean in mean filtering is dynamically adjusted to control the weight of mean filtering and intelligently control the effectiveness of hydrological data cleaning. This approach ensures the proper handling of outlier data while minimizing the erroneous processing of correct data.

[0221] In this disclosure, relevant hydrological data recorded by hydrological stations are obtained, and a slope difference filtering method suitable for hydrological elements is proposed to predict future data by combining recent data trends and to judge the rationality of current data. Then, a hydrological mean filtering algorithm is proposed, and the effect of the cleaning algorithm is intelligently adjusted according to data trends and waveforms to complete the storage of cleaned data and record abnormal data.

[0222] Figure 8 This is a schematic diagram of a hydrological element data cleaning and processing system provided in one embodiment of the present invention, as shown below. Figure 8 As shown, the system includes:

[0223] Acquisition module 11 is used to acquire hydrological data, including water level data.

[0224] The initial filtering module 12 is used to perform initial filtering on the hydrological data using a slope difference algorithm to obtain preliminary cleaning data;

[0225] The secondary filtering module 13 is used to filter the preliminary cleaning data again using the mean filtering algorithm to obtain the final cleaning data;

[0226] Storage module 14 is used to store the final cleaning data.

[0227] In one embodiment, the secondary filtering module 13 is specifically used for:

[0228] Obtain the fluctuation parameters of the preliminary cleaning data;

[0229] Determine whether the preliminary cleaning data is fluctuating based on the fluctuation parameters;

[0230] When the preliminary cleaned data is not the fluctuating data, the first mean filtering algorithm is used to clean the preliminary cleaned data to obtain the final cleaned data;

[0231] When the initial cleaning data is the fluctuation data, it is detected whether the fluctuation data is normal fluctuation data;

[0232] If the fluctuation data is normal fluctuation data, the second mean filtering algorithm is used to clean the preliminary cleaned data to obtain the final cleaned data.

[0233] If the fluctuation data is abnormal fluctuation data, the preliminary cleaned data is determined to be the final cleaned data.

[0234] In one embodiment, the initial filtering module 12 is specifically used for:

[0235] Obtain the change in two temporally adjacent hydrological data points from the hydrological data;

[0236] The time interval for obtaining two adjacent hydrological data points;

[0237] The slope of the two adjacent hydrological data points is obtained based on the change in the two adjacent hydrological data points and the corresponding time interval.

[0238] The target slope is obtained when the slope of two adjacent hydrological data points is greater than a preset slope threshold. The hydrological data points corresponding to the target slope are preliminary anomalies.

[0239] Obtain the water level difference between two adjacent preliminary anomaly points;

[0240] The target anomaly point is obtained when the water level difference is greater than a preset change threshold. The target anomaly point is the real anomaly data.

[0241] In one embodiment, the secondary filtering module 13 is specifically used for:

[0242] Obtain the hydrological data difference between any two time-series adjacent hydrological data in the preliminary cleaning data;

[0243] The fluctuation parameters of the preliminary cleaning data are obtained based on the number of all the preliminary cleaning data and the sum of the differences of all the hydrological data.

[0244] In one embodiment, the secondary filtering module 13 is specifically used for:

[0245] Detect whether the fluctuation parameter is greater than a preset fluctuation threshold;

[0246] If the fluctuation parameter is greater than the preset fluctuation threshold, then the preliminary cleaning data is determined to be the fluctuation data;

[0247] If the fluctuation parameter is not greater than the preset fluctuation threshold, then the preliminary cleaning data is determined not to be the fluctuation data.

[0248] In one embodiment, the secondary filtering module 13 is specifically used for:

[0249] Obtain the peak value of all peaks in the fluctuation data;

[0250] The sharpness of the corresponding wave peak is obtained based on the peak value.

[0251] The fluctuation data corresponding to the wave peak whose sharpness is greater than a preset sharpness threshold is identified as the abnormal fluctuation data.

[0252] The fluctuation data corresponding to the wave peak whose sharpness is not greater than the preset sharpness threshold is determined to be the normal fluctuation data.

[0253] In one embodiment, the secondary filtering module 13 is specifically used for:

[0254] For each of the stated peaks, perform the following steps:

[0255] Get a preset number of other peaks adjacent to the current peak;

[0256] Obtain the center of the current peak and the other peaks;

[0257] The sharpness of the current peak is obtained based on the peak value at the center of the peak and the peak value of the current peak.

[0258] In one embodiment, the secondary filtering module 13 is specifically used for:

[0259] Identify any abnormal data in the initial cleaned data;

[0260] The abnormal data is filtered out from the preliminary cleaning data to obtain the final cleaning data;

[0261] The process of identifying anomalous data in the preliminary cleaned data includes:

[0262] For each piece of cleaned data in the initial cleaned data, perform the following steps:

[0263] Get the time when the current cleaning data occurred;

[0264] Acquire a preset number of historical hydrological data, filtered by the slope difference algorithm, that are the same as the occurrence time.

[0265] Obtain the mean of the historical hydrological data;

[0266] Based on the mean and the current cleaning data, obtain the mean feature difference corresponding to the current cleaning data;

[0267] If the difference in the mean features is greater than a preset mean threshold, then the current cleaned data is determined to be abnormal data.

[0268] In one embodiment, the preset mean threshold corresponding to the first mean filtering algorithm is less than the preset mean threshold corresponding to the second mean filtering algorithm.

[0269] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0270] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for cleaning and processing hydrological element data, characterized in that, The method includes: The slope difference algorithm is used to perform initial filtering of hydrological data to obtain preliminary cleaned data. Obtain the fluctuation parameters of the preliminary cleaning data; Determine whether the preliminary cleaning data is fluctuating based on the fluctuation parameters; When the preliminary cleaned data is not the fluctuating data, the first mean filtering algorithm is used to clean the preliminary cleaned data to obtain the final cleaned data; When the initial cleaning data is the fluctuation data, it is detected whether the fluctuation data is normal fluctuation data; If the fluctuation data is normal fluctuation data, the second mean filtering algorithm is used to clean the preliminary cleaned data to obtain the final cleaned data. If the fluctuation data is abnormal fluctuation data, the preliminary cleaning data is determined to be the final cleaning data; The initial filtering of hydrological data using the slope difference algorithm to obtain preliminary cleaned data includes: Obtain the change in two temporally adjacent hydrological data points from the hydrological data; The time interval for obtaining two adjacent hydrological data points; The slope of the two adjacent hydrological data points is obtained based on the change in the two adjacent hydrological data points and the corresponding time interval. The target slope is obtained when the slope of two adjacent hydrological data points is greater than a preset slope threshold. The hydrological data points corresponding to the target slope are preliminary anomalies. Obtain the water level difference between two adjacent preliminary anomaly points; Identify target anomalies where the water level difference is greater than a preset change threshold, where the target anomalies are real anomaly data. The preliminary cleaning data is obtained by filtering the real abnormal data from the hydrological data; The fluctuation parameters for obtaining the preliminary cleaning data include: Obtain the hydrological data difference between any two time-series adjacent hydrological data in the preliminary cleaning data; The fluctuation parameters of the preliminary cleaning data are obtained based on the number of all the preliminary cleaning data and the sum of the differences of all the hydrological data.

2. The hydrological element data cleaning and processing method according to claim 1, characterized in that, After finalizing the cleaned data, the following also includes: Store the final cleaning data.

3. The hydrological element data cleaning and processing method according to claim 1, characterized in that, The step of determining whether the preliminary cleaning data is fluctuating data based on the fluctuation parameters includes: Detect whether the fluctuation parameter is greater than a preset fluctuation threshold; If the fluctuation parameter is greater than the preset fluctuation threshold, then the preliminary cleaning data is determined to be the fluctuation data; If the fluctuation parameter is not greater than the preset fluctuation threshold, then the preliminary cleaning data is determined not to be the fluctuation data.

4. The hydrological element data cleaning and processing method according to claim 3, characterized in that, The detection of whether the fluctuation data is normal fluctuation data includes: Obtain the peak value of all peaks in the fluctuation data; The sharpness of the corresponding wave peak is obtained based on the peak value. The fluctuation data corresponding to the wave peak whose sharpness is greater than a preset sharpness threshold is identified as the abnormal fluctuation data. The fluctuation data corresponding to the wave peak whose sharpness is not greater than the preset sharpness threshold is determined to be the normal fluctuation data.

5. The hydrological element data cleaning and processing method according to claim 4, characterized in that, The step of obtaining the sharpness corresponding to the peak value includes: For each of the stated peaks, perform the following steps: Get a preset number of other peaks adjacent to the current peak; Obtain the center of the current peak and the other peaks; The sharpness of the current peak is obtained based on the peak value at the center of the peak and the peak value of the current peak.

6. The hydrological element data cleaning and processing method according to claim 5, characterized in that, The step of cleaning the preliminary cleaned data using the first mean filtering algorithm includes: Identify any abnormal data in the initial cleaned data; The abnormal data is filtered out from the preliminary cleaning data to obtain the final cleaning data; The process of identifying anomalous data in the preliminary cleaned data includes: For each piece of cleaned data in the initial cleaned data, perform the following steps: Get the time when the current cleaning data was generated; Acquire a preset number of historical hydrological data, filtered by the slope difference algorithm, that are the same as the occurrence time. Obtain the mean of the historical hydrological data; Based on the mean and the current cleaning data, obtain the mean feature difference corresponding to the current cleaning data; If the difference in the mean features is greater than a preset mean threshold, then the current cleaned data is determined to be abnormal data.

7. The hydrological element data cleaning and processing method according to claim 6, characterized in that, The preset mean threshold corresponding to the first mean filtering algorithm is less than the preset mean threshold corresponding to the second mean filtering algorithm.

8. A hydrological element data cleaning and processing system, characterized in that, The system includes: The data acquisition module is used to collect hydrological data, including water level data. The initial filtering module is used to perform initial filtering on the hydrological data using a slope difference algorithm to obtain preliminary cleaning data. A secondary filtering module is used to acquire fluctuation parameters of the preliminary cleaned data; determine whether the preliminary cleaned data is fluctuating data based on the fluctuation parameters; if the preliminary cleaned data is not fluctuating data, use a first mean filtering algorithm to clean the preliminary cleaned data to obtain final cleaned data; if the preliminary cleaned data is fluctuating data, detect whether the fluctuating data is normal fluctuating data; if the fluctuating data is normal fluctuating data, use a second mean filtering algorithm to clean the preliminary cleaned data to obtain the final cleaned data; if the fluctuating data is abnormal fluctuating data, determine that the preliminary cleaned data is the final cleaned data. A storage module is used to store the final cleaning data; The initial filtering of hydrological data using the slope difference algorithm to obtain preliminary cleaned data includes: Obtain the change in two temporally adjacent hydrological data points from the hydrological data; The time interval for obtaining two adjacent hydrological data points; The slope of the two adjacent hydrological data points is obtained based on the change in the two adjacent hydrological data points and the corresponding time interval. The target slope is obtained when the slope of two adjacent hydrological data points is greater than a preset slope threshold. The hydrological data points corresponding to the target slope are preliminary anomalies. Obtain the water level difference between two adjacent preliminary anomaly points; Identify target anomalies where the water level difference is greater than a preset change threshold, where the target anomalies are real anomaly data. The preliminary cleaning data is obtained by filtering the real abnormal data from the hydrological data; The fluctuation parameters for obtaining the preliminary cleaning data include: Obtain the hydrological data difference between any two time-series adjacent hydrological data in the preliminary cleaning data; The fluctuation parameters of the preliminary cleaning data are obtained based on the number of all the preliminary cleaning data and the sum of the differences of all the hydrological data.

Images