Cascade hydropower station data cleaning method, system and equipment based on Pauta criterion and medium
Through the data cleaning method based on the Laida criteria, the problems of incomplete and inconsistent data of cascade hydropower stations were solved, and efficient data cleaning and quality improvement were achieved, ensuring that the data were suitable for subsequent analysis.
Patent Information
- Application Number
- CN202510771754.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-11
- Publication Date
- 2025-09-16
AI Technical Summary
Existing technologies are unable to effectively process incomplete and inconsistent dirty data from cascade hydropower stations, resulting in distorted data mining simulations. In particular, data types such as dam water level, unit output, and gate opening cannot be directly processed.
A data cleaning method based on the Raida criterion was adopted, including manual selection method, abnormal number ratio selection method and two-Raida criterion method. Through data processing and screening, the two-Raida criterion method was determined as the target method, data outliers were eliminated and interpolation was performed to complete data cleaning.
The quality and consistency of cascade hydropower station data are improved, outliers and missing values are reduced, and the data are ensured to be suitable for subsequent data mining and simulation calculations.
Smart Images
Figure CN120653639A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of water resource management, and in particular to a data cleaning method, system, equipment and medium for cascade hydropower stations based on the Laida criterion. Background Art
[0002] After being collected and used, the real-time operating data of important operating parameters of cascade hydropower stations are directly stored in the corresponding database. However, most of this data collected directly in the real world is incomplete, inconsistent, and dirty data, making it impossible to directly apply advanced data technologies such as data mining. To prevent simulation distortion caused by low historical data quality during the scheduling and operation of cascade hydropower stations using digital technology construction simulations, existing technologies usually carry out data governance work on a specific thematic data set. For example, most existing methods only consider the water level characteristics in front of the dam of cascade hydropower stations, but the system has many other types of data, such as the water level below the dam, the output of each unit, and the opening of each gate. These data types cannot be directly processed using the above methods. There is an urgent need for a method to further study the common problems of data collection. Summary of the Invention
[0003] The purpose of the present invention is to provide a data cleaning method, system, equipment and medium for cascade hydropower stations based on the Laida criterion, aiming to solve or improve at least one of the above-mentioned technical problems.
[0004] To achieve the above object, the present invention provides the following solutions:
[0005] A data cleaning method for cascade hydropower stations based on the Laida criterion, comprising:
[0006] Obtain historical data and preliminarily select a data cleaning method; the preliminarily selected data cleaning method includes manual selection method, abnormal number ratio selection method and two-Laida criterion method;
[0007] The historical data is processed using each of the data cleaning methods to obtain three calculation results, and the two-Laida criterion method is determined as the target data cleaning method based on the screening of the calculation results;
[0008] The target data cleaning method is used to process the target data, determine data outliers, remove and interpolate the data outliers, and complete the data cleaning of the cascade hydropower station.
[0009] Optionally, the processing of the manual selection method specifically includes:
[0010] Define the variable interval bias as the set value and initialize the initial value of the rolling data;
[0011] Perform 3σ dynamic detection on historical data within the specified interval (i, i+bias). After traversing all data, the iteration ends and the abnormal data is output; where i is 0 or a positive integer.
[0012] Optionally, the processing of the abnormal number ratio selection method specifically includes:
[0013] The outlier ratio a is defined as 0.0026, and the initial value is specified; the initial value includes an initial window length i of 5, an infinite value d as a set value, and the optimal value of i is 0;
[0014] Process all historical data, update the absolute value b1 of the abnormal value ratio and the target ratio, and determine whether the absolute value b1 is greater than the infinite value d: if not, assign the absolute value b1 to the infinite value d to complete the d value update; if so, the infinite value d uses the set value to complete the d value update;
[0015] The updated d value is used to calculate the optimal i, and the abnormal data is output when the set iteration conditions are met.
[0016] Optionally, the processing of the two-Laida criterion method specifically includes:
[0017] Perform first-order difference operation on two adjacent data in the historical data to obtain first-order difference data;
[0018] The first-order difference data and the original data in the historical data are processed respectively using the 3σ rule, and the first-order difference outliers and the original data outliers are determined by detailed screening. When any data is both a first-order difference outlier and an original data outlier, it is determined as the final outlier.
[0019] Optionally, the formula for the first-order difference operation is:
[0020] y n =H n+1 -H n
[0021] Among them, y n represents the flow difference at time point n, H n+1 represents the flow value at time point n+1, H n Represents the flow value at time point n.
[0022] The present invention also provides a data cleaning system for cascade hydropower stations based on the Laida criterion, comprising:
[0023] A preliminary method selection unit is used to obtain historical data and preliminarily select a data cleaning method; the preliminarily selected data cleaning method includes a manual selection method, an abnormal number ratio selection method, and a two-Laida criterion method;
[0024] a method screening and selection unit, configured to process the historical data using each of the data cleaning methods to obtain three calculation results, and to screen the results based on each of the calculation results to determine the two-Laida criterion method as the target data cleaning method;
[0025] The outlier cleaning unit is used to process the target data using the target data cleaning method, determine the data outliers, remove and interpolate the data outliers, and complete the data cleaning of the cascade hydropower station.
[0026] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above-mentioned cascade hydropower station data cleaning method based on the Laida criterion.
[0027] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned method for cleaning data of cascade hydropower stations based on the Laida criterion.
[0028] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0029] The present invention discloses a data cleaning method, system, device, and medium for cascade hydropower stations based on the Raida criterion. The method comprises acquiring historical data and preliminarily selecting a data cleaning method; the preliminarily selected data cleaning methods include a manual selection method, an outlier ratio selection method, and a two-Raida criterion method; processing the historical data using each of the data cleaning methods to obtain three calculation results, screening the calculation results based on each calculation, and determining the two-Raida criterion method as the target data cleaning method; processing the target data using the target data cleaning method to determine data outliers, and performing outlier removal and interpolation on the data outliers to complete data cleaning for the cascade hydropower station. The present invention can solve the problems of consistency in existing data sets and invalid and missing values. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0031] Figure 1 is the normal distribution frequency distribution histogram in this embodiment;
[0032] Figure 2 This is a flow chart of the manual selection method in this embodiment;
[0033] Figure 3 This is a flow chart of the method for selecting the proportion of abnormal numbers in this embodiment;
[0034] Figure 4 This is a schematic diagram of the two-step Laida criterion method flow chart in this embodiment;
[0035] Figure 5 This is a comparison diagram of the water level before and after treatment on the Shenxigou Dam in this embodiment;
[0036] Figure 6 This is a comparison diagram of the water level before and after the treatment of Shanan Dam in this embodiment;
[0037] Figure 7 Schematic diagram of the flow of the cascade hydropower station data cleaning method in this embodiment. DETAILED DESCRIPTION
[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0039] The purpose of the present invention is to provide a data cleaning method, system, equipment and medium for cascade hydropower stations based on the Laida criterion, aiming to solve or improve at least one of the above-mentioned technical problems.
[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0041] like Figure 1 As shown in FIG7 , the present invention provides a data cleaning method for cascade hydropower stations based on the Laida criterion, comprising:
[0042] Step 100: Acquire historical data and preliminarily select a data cleaning method; the preliminarily selected data cleaning method includes a manual selection method, an abnormal number ratio selection method, and a two-Laida criterion method.
[0043] Step 200: Process the historical data using the data cleaning methods to obtain three calculation results, and screen the results to determine the two-times Laida criterion method as the target data cleaning method.
[0044] Step 300: Process the target data using the target data cleaning method, determine data outliers, remove and interpolate the data outliers, and complete the cascade hydropower station data cleaning.
[0045] As a specific implementation method, the processing process of the above steps is provided.
[0046] The technical problem to be solved by this embodiment is to address the defects in data processing of the existing cascade hydropower station water condition data monitoring system, and provide a data cleaning method to reduce data errors, fill missing data, and improve data quality. This method can handle the problems of data set consistency and invalid and missing values.
[0047] To achieve the above objectives, the present invention is mainly divided into two parts to perform data cleaning processing on the data.
[0048] ①Data consistency check
[0049] Consistency checks examine data compliance based on the reasonable range of values and interrelationships for each variable. Any data outside the normal range, that is logically unreasonable, or that is contradictory should be identified. For example, when examining historical water level data, any negative values or values outside the reservoir's operating range should be considered outside the normal range. When inconsistencies are discovered, the questionnaire number, record number, variable name, and error category should be listed to facilitate further verification and correction.
[0050] ②Handling of invalid and missing values
[0051] Due to survey, coding and entry errors, there may be some invalid values and missing values in the data, which need to be properly handled. Multiple methods can be used to deal with abnormal data, such as the estimation method, which is to replace invalid values and missing values with the sample mean, median or mode of a certain variable. The whole case deletion method, which is to eliminate samples with missing values. The variable deletion method, if a variable has a lot of invalid values and missing values, and the variable is not particularly important for the research problem, you can consider deleting the variable. The paired deletion method, using a special code (usually 9, 99, 999, etc.) to represent invalid values and missing values, while retaining all variables and samples in the data set.
[0052] The Laida criterion, also known as the 3σ criterion, assumes that a set of test data contains only random errors. The standard deviation is then calculated and an interval is determined based on a certain probability. Errors exceeding this interval are considered gross errors, not random errors. Data containing these errors should be eliminated. This discriminant processing principle and method is limited to processing sample data with a normal or nearly normal distribution. It assumes that the number of measurements is sufficiently large. Using this criterion to eliminate gross errors is unreliable when the number of measurements is small.
[0053] like Figure 1The figure below shows a frequency distribution histogram for a normal distribution, where σ represents the standard deviation and μ represents the mean. x = μ is the axis of symmetry for the graph. The 3σ principle states that the probability of a value being in the interval (μ - σ, μ + σ) is 0.6826; the probability of a value being in the interval (μ - 2σ, μ + 2σ) is 0.9545; and the probability of a value being in the interval (μ - 3σ, μ + 3σ) is 0.9973. It can be assumed that the values of Y are almost entirely concentrated within the interval (μ - 3σ, μ + 3σ), with the probability of being outside this range being less than 0.3%.
[0054] The present invention proposes three methods based on the Laida criterion for different data types and forms, as follows:
[0055] Method 1: Manual selection
[0056] Execute steps such as Figure 2 As shown:
[0057] Select a variable interval; initialize the initial value of the rolling data; select data in the specified interval for rolling detection; if the data has not been detected, repeat the previous step; end.
[0058] Method 2: Selection method based on the proportion of abnormal numbers
[0059] Since manually selecting the data scale may result in too much or too little abnormal data (in the 3σ rule, the abnormal data ratio should be around 0.0026), manually modifying parameters in the code and re-running the code to generate data is too cumbersome, and some data segments may not obey the normal distribution, the abnormal data ratio is regarded as the most important factor. By specifying different data ratios, different data generate different data processing window lengths. The processing steps are as follows: Figure 3 As shown:
[0060] Specify the outlier ratio; specify the initial value; process all data and update the outlier ratio; calculate the optimal number; determine whether the end condition is met, otherwise return to the outlier ratio update step; end.
[0061] Method 3: Two-time Laida Criteria
[0062] Since the data changes are not significant after the data scale is unified, all the data are plotted. Figure 1 After a review, it is found that there are no more than ten outlier points in the six groups of data. Then, filtering outliers according to the proportion of abnormal data will filter out normal values. That is, the proportion of abnormal data should not be used as the most important screening condition. The processing steps are as follows Figure 4 As shown:
[0063] First, calculate the difference between two adjacent data points, and process the difference using the 3σ rule; the outliers initially screened out are then screened again to see if their neighbors are also outliers and their sum is approximately equal to zero. After two steps of screening, the differential outlier value is output; the original data is processed using the 3σ rule; the results of the two processings are combined; and the process ends.
[0064] As a specific embodiment, taking a cascade hydropower station in a river basin in southwest China as an example, the scheme is as follows:
[0065] Data: Water level data for the Pubugou, Shenxigou, Gongzui, and Tongjiezi dams over the past eleven years; Zhentouba over the past seven years; and Shannan over the past five years, on an hourly scale. See Table 1.
[0066] Table 1 Data details
[0067]
[0068]
[0069] Method 1 result analysis:
[0070] Due to the excessive amount of water level data, the box plot and 3σ static detection methods cannot detect outliers. Therefore, this outlier analysis uses the 3σ dynamic detection method. The length of data selected each time is set to 200 days (adjustable) for forward rolling detection.
[0071] Here, we first use the 3σ method for processing, followed by manual identification. Table 2 shows the statistical results of Data 2 after processing with Method 1, and Table 3 shows the specific situation of the "Final Anomaly" column in Table 2.
[0072] Table 2 Statistics of data processed by method 1
[0073]
[0074] Table 3 Specific circumstances of the final abnormality
[0075]
[0076]
[0077] Method 2 result analysis:
[0078] Manually specifying the window length may cause the following problems:
[0079] ① The proportion of abnormal data is too large or too small (in the 3σ rule, the proportion of abnormal data should be around 0.0026) ② Manually modifying parameters in the code and re-running the code to generate data is too cumbersome
[0080] ③Some data segments may not obey the normal distribution
[0081] The proportion of abnormal data is regarded as the most important factor. By stipulating different data proportions, different data generate different data processing window lengths.
[0082] Figure 5-Figure 6 The red dots in the middle are the abnormal data that were screened out, and two processing results were selected for analysis.
[0083] Treatment result 1: The water level on Shenxigou Dam is as follows Figure 5 As shown in the second treatment result, the water level on Shannan Dam is as follows Figure 6 shown.
[0084] It can be seen that this improvement is more reasonable in terms of the proportion of outliers.
[0085] Method 3 result analysis:
[0086] Calculate the first-order difference, first-order difference y n =H n+1 -H n , is the water level at the next moment minus the water level at the current moment. Since the calculation methods for first-order differences and flow differences are identical, the first-order difference images and flow difference images should be very similar under normal data conditions. After processing the first-order differences and the original data using the 3σ rule, and then applying some detailed screening, if the same data is identified as an outlier in both the original data and the first-order difference data, it is ultimately identified as an outlier. Specific statistics are shown in Table 4.
[0087] Table 4. The presence of outliers in the three screenings of six reservoirs
[0088]
[0089] The analysis shows that after applying the 3σ rule twice, we can accurately locate some "quasi-outlier points." By comparing these "quasi-outlier points" with actual inflow and outflow traffic, the resulting outliers are credible, as shown in Table 5.
[0090] Table 5 Statistics of the final abnormal total amount of the six reservoirs
[0091]
[0092] Therefore, this technical solution has the following beneficial effects:
[0093] The data processing method of this embodiment can convert data into minute-scale data, identify, filter, and clean outliers in the data, thereby improving data quality. After cleaning, the proportion of outliers in the data is very small, less than 0.25%, and it can be considered to be essentially free of outliers. This can be used for subsequent calculations.
[0094] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0095] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.
Claims
1. A data cleaning method for cascade hydropower stations based on the Laida criterion, characterized in that: include: Obtain historical data and preliminarily select a data cleaning method; the preliminarily selected data cleaning method includes manual selection method, abnormal number ratio selection method and two-Laida criterion method; The historical data is processed using each of the data cleaning methods to obtain three calculation results, and the two-Laida criterion method is determined as the target data cleaning method based on the screening of the calculation results; The target data cleaning method is used to process the target data, determine data outliers, remove and interpolate the data outliers, and complete the data cleaning of the cascade hydropower station.
2. The data cleaning method for cascade hydropower stations based on the Laida criterion according to claim 1 is characterized in that: The processing process of the manual selection method specifically includes: Define the variable interval bias as the set value and initialize the initial value of the rolling data; Perform 3σ dynamic detection on historical data within the specified interval (i, i+bias). After traversing all data, the iteration ends and the abnormal data is output; where i is 0 or a positive integer.
3. The data cleaning method for cascade hydropower stations based on the Laida criterion according to claim 1 is characterized in that: The processing process of the abnormal number ratio selection method specifically includes: The outlier ratio a is defined as 0.0026, and the initial value is specified; the initial value includes an initial window length i of 5, an infinite value d as a set value, and the optimal value of i is 0; Process all historical data, update the absolute value b1 of the abnormal value ratio and the target ratio, and determine whether the absolute value b1 is greater than the infinite value d: if not, assign the absolute value b1 to the infinite value d to complete the d value update; if so, the infinite value d uses the set value to complete the d value update; The updated d value is used to calculate the optimal i, and the abnormal data is output when the set iteration conditions are met.
4. The data cleaning method for cascade hydropower stations based on the Laida criterion according to claim 1 is characterized in that: The two-step Laida process specifically includes: Perform first-order difference operation on two adjacent data in the historical data to obtain first-order difference data; The first-order difference data and the original data in the historical data are processed respectively using the 3σ rule, and the first-order difference outliers and the original data outliers are determined by detailed screening. When any data is both a first-order difference outlier and an original data outlier, it is determined as the final outlier.
5. The data cleaning method for cascade hydropower stations based on the Laida criterion according to claim 4 is characterized in that: The formula for the first-order difference operation is: y n =H n+1 -H n Among them, y n represents the flow difference at time point n, H n+1 represents the flow value at time point n+1, H n Represents the flow value at time point n.
6. A data cleaning system for cascade hydropower stations based on the Laida criterion, characterized in that: include: A preliminary method selection unit is used to obtain historical data and preliminarily select a data cleaning method; The data cleaning methods initially selected include manual selection method, abnormal number ratio selection method and two-times Laida criterion method; a method screening and selection unit, configured to process the historical data using each of the data cleaning methods to obtain three calculation results, and to screen the results based on each of the calculation results to determine the two-Laida criterion method as the target data cleaning method; The outlier cleaning unit is used to process the target data using the target data cleaning method, determine the data outliers, remove and interpolate the data outliers, and complete the data cleaning of the cascade hydropower station.
7. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the cascade hydropower station data cleaning method based on the Laida criterion according to any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that It stores a computer program, which, when executed by a processor, implements the cascade hydropower station data cleaning method based on the Laida criterion as described in any one of claims 1 to 5.