Hydrological Data Cleaning Method and System for Hydrological Stations Based on Big Data Analysis

Abstract

The present invention relates to the technical field of big data analysis, and particularly to a method and system for cleaning hydrological data of a hydrological station based on big data analysis. The present invention judges the error of hydrological flow data based on the change of hydrological flow data at the basin location corresponding to the time and the change of upstream and downstream water level data, evaluates the abnormality of the hydrological data terminal through the environment of the acquisition point and the operation data of the acquisition equipment, obtains the error judgment result of the hydrological flow data and the abnormality evaluation result of the hydrological data terminal to judge the authenticity of the hydrological flow data, and cleans the hydrological flow data according to the authenticity judgment result of the hydrological flow data to obtain the cleaned hydrological flow data. Based on the comprehensive analysis of hydrological data error and the abnormality evaluation of the hydrological data terminal, the authenticity of the obtained hydrological flow data is accurately judged, and the accuracy and security of data cleaning are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data analysis, and particularly to a method and system for cleaning hydrological data of a hydrological station based on big data analysis. Background Art

[0002] When monitoring hydrological flow at a hydrological station, a large amount of hydrological data is generated. However, due to errors in hydrological equipment or equipment failures, this hydrological data contains a large amount of incorrect data such as unrealistic fluctuations or mutations. At this time, it is necessary to clean the hydrological data of the hydrological station. The existing method calculates the water level slope and difference based on the water level data and time of two adjacent times, and the algorithm makes judgments in two layers, the outer layer for preliminary filtering and the inner layer for advanced filtering. The outer layer has a strict difference and a loose slope, and the inner layer has a strict slope and a loose difference. The purpose is to filter abnormal data with large sudden change amounts and high slopes, using a filtering method with the slope as the main and the difference as the auxiliary to ensure the accuracy of abnormal data processing while avoiding misprocessing of normal data. However, it cannot comprehensively analyze based on hydrological data errors and anomalies at the hydrological data terminal, and the accuracy of judging the authenticity of the obtained hydrological flow data is not high. The prior art, for example, discloses a hydrological data cleaning method and system in a Chinese patent with the authorization announcement number CN113377750B, which obtains the hydrological data to be processed; performs the first cleaning on the hydrological data to be processed; the first cleaning includes: missing data monitoring and elimination and filling of unreasonable data; performs the second cleaning on the hydrological data after the first cleaning; the second cleaning includes: elimination and filling of data with inconsistent causal relationships; and checks the cleaning result. The multi - layer quadratic cleaning of hydrological data can provide reasonable and consistent data for further hydrological research. The prior art has the technical problems proposed in this application.

[0003] To solve the technical problems proposed in this application, the present application designs a method and system for cleaning hydrological data of a hydrological station based on big data analysis. Summary of the Invention

[0004] In order to overcome the defects and deficiencies existing in the prior art, the present invention provides a method and system for cleaning hydrological data of a hydrological station based on big data analysis.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] In the first aspect, the present invention provides a method for cleaning hydrological data of a hydrological station based on big data analysis, including the following steps:

[0007] S1. Obtain the hydrological flow data at each time of the basin location collected by the hydrological station, and at the same time obtain the environment of the collection point and the operation data of the collection equipment;

[0008] S2. Perform error judgment on the hydrological flow data based on the changes in the hydrological flow data at the basin location corresponding to the time and the changes in the upstream and downstream water level data;

[0009] S3. Conduct abnormal assessment of the hydrological data terminal through the obtained environment of the collection points and the operation data of the collection equipment;

[0010] S4. Obtain the error judgment result of the hydrological flow data and the abnormal assessment result of the hydrological data terminal to judge the authenticity of the hydrological flow data;

[0011] S5. Clean the hydrological flow data according to the authenticity judgment result of the hydrological flow data to obtain the cleaned hydrological flow data.

[0012] In an implementation manner of the present invention, the obtaining of the hydrological data at each time of the basin location collected by the hydrological station, and at the same time obtaining the environment of the collection points and the operation data of the collection equipment includes the following specific steps:

[0013] S101. The hydrological data terminals installed at each monitoring point in the basin collect the hydrological data of each monitoring point, construct a change curve of the hydrological flow data of each monitoring point changing with time, and store the obtained change curves of each monitoring point in the corresponding storage module;

[0014] S102. Obtain the weather environment data and electromagnetic field environment data of the monitoring location corresponding to each monitoring point, and store them in the corresponding storage component. Since the weather environment and electromagnetic field environment of the monitoring location will have a negative impact on the accuracy of the collection terminal;

[0015] S103. Collect the operation data of the hydrological data terminal through the operation data collection terminal. Among them, the operation data includes the operation time, the frequency of collecting data, and the fluctuation of the collected data, and store them in the corresponding storage component, so as to comprehensively analyze the aging probability of the hydrological data terminal.

[0016] In an implementation manner of the present invention, in step S2, the error judgment of the hydrological flow data based on the changes in the hydrological flow data at the basin location corresponding to the time and the changes in the upstream and downstream water level data includes the following specific steps:

[0017] S201. Obtain the hydrological flow data of the monitoring location at each monitoring time point within the corresponding period, and at the same time obtain the hydrological flow data of the upstream and downstream monitoring locations corresponding to the monitoring location, and at the same time obtain the weather environment data within the monitoring time interval;

[0018] S202. Construct a neural network model based on the hydrological flow data of the upstream and downstream monitoring positions corresponding to the historical period, the weather environment within the time interval of the upstream and downstream monitoring positions corresponding to the historical corresponding monitoring position, and the hydrological flow data of the corresponding monitoring position in the historical data period to estimate the predicted value of the hydrological flow data at each monitoring time point within the corresponding period.

[0019] S203. Obtain the predicted value of the hydrological flow data at each monitoring time point within the corresponding period and the actual value of the hydrological flow data at each monitoring time point within the corresponding period. Import the actual value and the predicted value of the hydrological flow data at the corresponding monitoring time point into the error judgment value calculation formula to calculate the error judgment value. The flow fluctuation is very small within a very short period of time. If there is a large fluctuation, it indicates that the measurement accuracy of the device is relatively low.

[0020] S204. Compare the calculated error judgment value of the corresponding monitoring position with the set error judgment threshold. If the error judgment value of the monitoring position is greater than or equal to the set error judgment threshold, set the corresponding monitoring position as an error position and perform S3. If the error judgment value of the monitoring position is less than the set error judgment threshold, it is determined that the data is normal and does not need to be cleaned.

[0021] In an implementation manner of the present invention, in step S3, the abnormal evaluation of the hydrological flow data is performed by obtaining the environment of the acquisition point and the operation data of the acquisition device, including the following specific steps:

[0022] S301. Obtain the environment of the acquisition point at the error position and the operation data of the hydrological data terminal.

[0023] S302. Perform an abnormal impact evaluation of the hydrological data terminal at the acquisition moment based on the environmental impact at the acquisition moment of the acquisition point at the error position and the operation data impact of the hydrological data terminal.

[0024] Among them, the abnormal impact evaluation of the hydrological data terminal includes the following specific steps:

[0025] S3021. Compare and analyze the environmental data at the acquisition moment of the acquisition point at the error position with the normal operation data of the hydrological data terminal to obtain the environmental abnormal impact result.

[0026] S3022. Analyze the impact of the operation data of the hydrological data terminal based on the historical operation data of the hydrological data terminal. Since the fluctuation of the collected data will affect the stability of the device, frequent data collection fluctuations will cause device wear.

[0027] S3023. Obtain the environmental abnormal impact result and the analysis result of the impact of the operation data of the hydrological data terminal, and perform weighted summation to obtain the abnormal impact evaluation result of the hydrological data terminal.

[0028] In an implementation manner of the present invention, the specific steps for determining the authenticity of hydrological flow data by obtaining the error judgment result of the hydrological flow data and the abnormal evaluation result of the hydrological data terminal in step S4 are as follows:

[0029] S401. Obtain the error judgment value corresponding to the hydrological flow data and the abnormal influence evaluation result of the hydrological data terminal at the time corresponding to the hydrological flow data;

[0030] S402. Obtain the abnormal value of the hydrological flow data by weighted summation after standardizing the error judgment value corresponding to the hydrological flow data and the abnormal influence evaluation result of the hydrological data terminal respectively, and obtain the authenticity result of the hydrological flow data by taking the reciprocal of the abnormal value of the hydrological flow data.

[0031] In an implementation manner of the present invention, the specific content of cleaning the hydrological flow data according to the authenticity judgment result of the hydrological flow data in step S5 to obtain the cleaned hydrological flow data is as follows:

[0032] Compare the obtained authenticity result of the corresponding hydrological flow data with the set authenticity threshold of the hydrological flow data. If the obtained authenticity result of the corresponding hydrological flow data is greater than or equal to the set authenticity threshold of the hydrological flow data, it means that the corresponding hydrological flow data is true and does not need to be cleaned; if the obtained authenticity result of the corresponding hydrological flow data is less than the set authenticity threshold of the hydrological flow data, it means that the corresponding hydrological flow data is not true and needs to be cleaned; perform cleaning on the hydrological flow data to obtain the cleaned hydrological flow data.

[0033] In a second aspect, the present invention also provides a hydrological station hydrological data cleaning system based on big data analysis, which specifically includes:

[0034] A data acquisition module, which is used to acquire the hydrological flow data of each time at the basin location collected by the hydrological station, and at the same time acquire the environment of the collection point and the operation data of the collection equipment; an error judgment module, which judges the error of the hydrological flow data based on the change of the hydrological flow data at the basin location corresponding to the time and the change of the upstream and downstream water level data; an abnormal evaluation module, which is used to evaluate the abnormality of the hydrological data terminal by acquiring the environment of the collection point and the operation data of the collection equipment; a data authenticity judgment module, which obtains the error judgment result of the hydrological flow data and the abnormal evaluation result of the hydrological data terminal to judge the authenticity of the hydrological flow data; a cleaning module, which cleans the hydrological flow data according to the authenticity judgment result of the hydrological flow data to obtain the cleaned hydrological flow data.

[0035] In a third aspect, an electronic device provided by the present invention includes: a processor and a memory. Among them, a computer program that can be called by the processor is stored in the memory, and the processor executes a method for cleaning hydrological data of a hydrological station based on big data analysis by calling the computer program stored in the memory.

[0036] In a fourth aspect, a computer-readable storage medium provided by the present invention stores instructions. When the instructions run on a computer, the computer is made to execute a method for cleaning hydrological data of a hydrological station based on big data analysis.

[0037] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0038] The present invention judges the error of hydrological flow data based on the change of hydrological flow data at the basin location corresponding to the time and the change of upstream and downstream water level data, evaluates the abnormality of the hydrological data terminal through the environment of the acquisition point and the operation data of the acquisition equipment, obtains the error judgment result of the hydrological flow data and the abnormality evaluation result of the hydrological data terminal to judge the authenticity of the hydrological flow data, cleans the hydrological flow data according to the authenticity judgment result of the hydrological flow data to obtain the cleaned hydrological flow data, and accurately judges the authenticity of the obtained hydrological flow data based on the comprehensive analysis of hydrological data error and the abnormality evaluation of the hydrological data terminal, improving the accuracy and security of data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objects and advantages of the present invention will become more apparent:

[0040] Figure 1 It is a schematic diagram of the overall flow of the method embodiment of the present invention;

[0041] Figure 2 It is a flowchart of error judgment of the method embodiment of the present invention;

[0042] Figure 3 It is a flowchart of evaluating the influence of abnormality of the hydrological data terminal in the method embodiment of the present invention;

[0043] Figure 4 It is a schematic diagram of the structure of the system embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] The technical solution of the present invention will be described in detail below through the drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. Without conflict, the technical features in the embodiments of the present invention and the embodiments can be combined with each other.

[0045] Example 1

[0046] Example 1 provides a method embodiment of the technical solution.

[0047] As Figures 1 to 3 shown, this embodiment provides a method for cleaning hydrological data of a hydrological station based on big data analysis, specifically including the following steps:

[0048] S1. Obtain the hydrological flow data at each time of the basin location collected by the hydrological station, and at the same time obtain the environment of the collection point and the operation data of the collection equipment;

[0049] In this embodiment, step S1 includes the following specific steps:

[0050] S101. The hydrological data terminals installed at each monitoring point in the basin collect the hydrological data of each monitoring point, construct a change curve of the hydrological flow data of each monitoring point over time, and store the obtained change curves of each monitoring point in the corresponding storage module;

[0051] S102. Obtain the weather environment data and electromagnetic field environment data of the monitoring location corresponding to each monitoring point, and store them in the corresponding storage component. Since the weather environment and electromagnetic field environment of the monitoring location will have a negative impact on the accuracy of the collection terminal;

[0052] S103. Collect the operation data of the hydrological data terminal through the operation data collection terminal. Among them, the operation data includes the operation time, the frequency of collecting data, and the fluctuation of the collected data, and store it in the corresponding storage component, so as to comprehensively analyze the aging probability of the hydrological data terminal

[0053] S2. Based on the change of the hydrological flow data at the basin location corresponding to the time and the change of the upstream and downstream water level data, judge the error of the hydrological flow data;

[0054] In step S2, based on the change of the hydrological flow data at the basin location corresponding to the time and the change of the upstream and downstream water level data, judge the error of the hydrological flow data, including the following specific steps:

[0055] S201. Obtain the hydrological flow data of the monitoring location at each monitoring time point within the corresponding period, and at the same time obtain the hydrological flow data of the upstream and downstream monitoring locations corresponding to the monitoring location, and at the same time obtain the weather environment data within the monitoring time interval;

[0056] S202. Based on the hydrological flow data of the upstream and downstream monitoring positions corresponding to the historical period, the weather environment within the time interval of the upstream and downstream monitoring positions corresponding to the historical corresponding monitoring position, and the hydrological flow data of the corresponding monitoring position in the historical data period, construct a neural network model to estimate the predicted value of the hydrological flow data of the monitoring position at each monitoring time point within the corresponding period. The construction process of the neural network model includes the following specific steps: collect the historical hydrological flow data of each monitoring point (upstream, downstream, target point), and obtain the upstream and downstream weather environment data corresponding to the historical data period; divide it into a training set (such as the first 80%), a validation set (10%), and a test set (10%) in chronological order: according to the data characteristics and complexity, select a suitable neural network model, such as combining the historical flows of the LSTM target point, upstream and downstream points, plus weather data, to form an input feature vector, and the output is the predicted flow value at each future time point. Select the Adam optimizer and the mean square error loss function; perform model training and use the early stopping method to prevent overfitting; use grid search or random search to optimize model parameters, such as learning rate, number of hidden layers, batch size, etc.; use indicators such as MAE, RMSE, and R2 to evaluate the model performance on the validation set and the test set; perform cross-validation of time series segmentation to ensure the stability and generalization ability of the model; compare the prediction results of different models and select the model with the best performance. The specific content of the construction method of the neural network model is the conventional technology of those skilled in the art and will not be elaborated in detail here;

[0057] S203. Obtain the predicted value of the hydrological flow data of the monitoring position at each monitoring time point within the corresponding period and the actual value of the hydrological flow data of the monitoring position at each monitoring time point within the corresponding period, and import the actual value and the predicted value of the hydrological flow data of the monitoring position at the corresponding time point into the error judgment value calculation formula to calculate the error judgment value. The error judgment value calculation formula for the monitoring position at the corresponding time point is: , where \(x_t\) is the actual value of the hydrological flow data of the monitoring position at the corresponding time point, \(x_{ct}\) is the predicted value of the hydrological flow data of the monitoring position at the corresponding time point, \(x_{(t - 1)}\) is the actual value of the hydrological flow data of the previous monitoring position, \(x_m\) is the flow fluctuation safety value, is the error proportion weight, is the fluctuation proportion weight. In this formula, the former formula is the error judgment between the actual value and the predicted value, and the latter formula is the error judgment of data fluctuation. The flow fluctuation is very small in a very short time. If there is a large fluctuation, it means that the measurement accuracy of the equipment is relatively low;

[0058] S204. Compare the calculated monitoring position error judgment value at the corresponding time point with the set error judgment threshold. If the monitoring position error judgment value is greater than or equal to the set error judgment threshold, set the corresponding monitoring position as the error position and proceed to S3. If the monitoring position error judgment value is less than the set error judgment threshold, it is determined that the data is normal and no cleaning is required.

[0059] S3. Conduct an abnormal assessment of the hydrological data terminal through the obtained environment of the collection points and the operation data of the collection equipment.

[0060] In this embodiment, the abnormal assessment of the hydrological flow data in step S3 through the obtained environment of the collection points and the operation data of the collection equipment includes the following specific steps:

[0061] S301. Obtain the environment of the collection points at the error position and the operation data of the hydrological data terminal.

[0062] S302. Conduct an abnormal impact assessment of the hydrological data terminal at the collection time based on the environmental impact at the collection time of the collection points at the error position and the operation data impact of the hydrological data terminal.

[0063] Among them, the abnormal impact assessment of the hydrological data terminal includes the following specific steps:

[0064] S3021. Conduct a comparative analysis between the environmental data at the collection time of the collection points at the error position and the normal operation data of the hydrological data terminal to obtain the environmental abnormal impact result. Among them, the calculation formula of the environmental abnormal impact result is: , where N is the type of environmental impact, ci is the impact factor of the i-th type of environmental impact. Here, the environmental impact is the type of environment that affects the collection accuracy of the hydrological data terminal, such as environmental temperature, humidity, etc., kit is the specific value of the i-th type of environment at the corresponding collection time, and kim is the median value of the operation safety range of the i-th type of environment.

[0065] S3022. Conduct an analysis of the operation data impact of the hydrological data terminal based on the historical operation data of the hydrological data terminal. Among them, the calculation formula for the analysis of the operation data impact of the hydrological data terminal is: , where tc is the operation duration of the hydrological data terminal, tm is the designed life of the hydrological data terminal, Tz is the data collection frequency of the hydrological data terminal, Tc is the maximum value of the data collection frequency safety range of the hydrological data terminal, exp() is the exponential power of the natural constant e, m is the number of data collections of the hydrological data terminal, zj is the collection data of the j-th collection, z(j - 1) is the collection data of the (j - 1)-th collection, and zs is the range value of the measurement range of the hydrological data terminal, that is, the maximum value of the measurement range minus the minimum value. Since the fluctuation of the collection data will affect the stability of the equipment, frequent data collection fluctuations will cause equipment wear.

[0066] S3023. Obtain the abnormal influence result of the environment and the operation data influence analysis result of the hydrological data terminal, and perform weighted summation to obtain the abnormal influence evaluation result of the hydrological data terminal;

[0067] S4. Obtain the error judgment result of the hydrological flow data and the abnormal evaluation result of the hydrological data terminal to judge the authenticity of the hydrological flow data;

[0068] In this embodiment, obtaining the error judgment result of the hydrological flow data and the abnormal evaluation result of the hydrological data terminal in step S4 to judge the authenticity of the hydrological flow data includes the following specific steps:

[0069] S401. Obtain the error judgment value corresponding to the hydrological flow data and the abnormal influence evaluation result of the hydrological data terminal at the moment corresponding to the hydrological flow data;

[0070] S402. Respectively standardize the error judgment value corresponding to the hydrological flow data and the abnormal influence evaluation result of the hydrological data terminal, and then perform weighted summation to obtain the abnormal value of the hydrological flow data. Take the reciprocal of the abnormal value of the hydrological flow data to obtain the authenticity result of the hydrological flow data. Among them, the calculation result of the authenticity result of the hydrological flow data can be: , where Wc is the abnormal influence evaluation result of the hydrological data terminal, is the proportion weight of the error judgment value, is the proportion weight of the abnormal influence evaluation result of the terminal;

[0071] S5. Clean the hydrological flow data according to the authenticity judgment result of the hydrological flow data to obtain the cleaned hydrological flow data;

[0072] In this embodiment, cleaning the hydrological flow data according to the authenticity judgment result of the hydrological flow data in step S5 to obtain the cleaned hydrological flow data includes the following specific contents:

[0073] Compare the obtained authenticity result of the corresponding hydrological flow data with the set authenticity threshold of the hydrological flow data. If the obtained authenticity result of the corresponding hydrological flow data is greater than or equal to the set authenticity threshold of the hydrological flow data, it means that the corresponding hydrological flow data is true and does not need to be cleaned; if the obtained authenticity result of the corresponding hydrological flow data is less than the set authenticity threshold of the hydrological flow data, it means that the corresponding hydrological flow data is not true and needs to be cleaned; perform cleaning on the hydrological flow data to obtain the cleaned hydrological flow data.

[0074] It should be noted that in this embodiment, the method for obtaining the set parameters (such as the proportion weights of each weighted sum, thresholds, etc.) in this embodiment is obtained through experiments by those skilled in the art. The specific experimental method is as follows: Obtain the hydrological flow data of each time at the basin location collected by several historical hydrological stations. At the same time, obtain the environment of the collection points and the operation data of the collection equipment historically, substitute them into each step of this embodiment to calculate the authenticity results of the corresponding hydrological flow data. At the same time, obtain the judgment results of on-site detection of whether the hydrological flow is real. Based on the authenticity results of the corresponding hydrological flow data and the judgment results of on-site detection of whether the hydrological flow is real, import them into a fitting software (preferably matlab) for iterative fitting of the data, and output the set parameter values that meet the maximum judgment accuracy rate.

[0075] It should be noted that in this embodiment, the following advantages exist. The error of the hydrological flow data is judged based on the change of the hydrological flow data at the basin location corresponding to the time and the change of the upstream and downstream water level data. The abnormal evaluation of the hydrological data terminal is carried out by obtaining the environment of the collection points and the operation data of the collection equipment. The authenticity of the hydrological flow data is judged by obtaining the error judgment results of the hydrological flow data and the abnormal evaluation results of the hydrological data terminal. The hydrological flow data is cleaned according to the authenticity judgment results of the hydrological flow data to obtain the cleaned hydrological flow data. Based on the comprehensive analysis of the hydrological data error and the abnormal evaluation of the hydrological data terminal, the authenticity of the obtained hydrological flow data is accurately judged, improving the accuracy and security of data cleaning.

[0076] Embodiment 2

[0077] As Figure 4 shown, this embodiment provides a hydrological data cleaning system for a hydrological station based on big data analysis, including:

[0078] A data acquisition module is used to acquire the hydrological flow data of each time at the basin location collected by a hydrological station, and at the same time acquire the environment of the collection point and the operation data of the collection equipment; an error judgment module is used to judge the error of the hydrological flow data based on the change of the hydrological flow data at the basin location at the corresponding time and the change of the upstream and downstream water level data; an anomaly assessment module is used to conduct an anomaly assessment of the hydrological data terminal through the acquired environment of the collection point and the operation data of the collection equipment; a data authenticity judgment module is used to obtain the error judgment result of the hydrological flow data and the anomaly assessment result of the hydrological data terminal to judge the authenticity of the hydrological flow data; a cleaning module is used to clean the hydrological flow data according to the authenticity judgment result of the hydrological flow data to obtain the cleaned hydrological flow data. For the parameters and the steps for each unit module in the above-mentioned hydrological station hydrological data cleaning system based on big data analysis of the present invention to achieve corresponding functions and the corresponding roles, reference can be made to the parameters and steps in the embodiments of the hydrological station hydrological data cleaning method based on big data analysis in the above text, which will not be elaborated here.

[0079] Embodiment 3

[0080] An electronic device according to an embodiment of the present invention includes: a processor and a memory. Among them, a computer program that can be called by the processor is stored in the memory, and the processor executes the hydrological station hydrological data cleaning method based on big data analysis by calling the computer program stored in the memory. It should be noted that: all computer programs of the hydrological station hydrological data cleaning method based on big data analysis are implemented using the C language.

[0081] Embodiment 4

[0082] This embodiment proposes a computer-readable storage medium, on which a rewritable computer program is stored;

[0083] When the computer program runs on a computer device, the computer device is enabled to execute the above-mentioned hydrological station hydrological data cleaning method based on big data analysis.

[0084] Each embodiment in the present invention is described in a progressive manner. For the same or similar parts between each embodiment, reference can be made to each other. The key point of each embodiment is to illustrate the differences from other embodiments. In particular, for the embodiments of the Internet of Things devices and media, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments.

[0085] The systems and media provided by the embodiments of the present invention correspond one-to-one with the methods. Therefore, the systems and media also have beneficial technical effects similar to the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be elaborated here.

[0086] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0087] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or the combination of blocks.

[0088] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in Figure 1 one or more of the flows Figure 1 or blocks or the combination of blocks.

[0089] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0090] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory such as read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0091] A computer-readable medium includes both permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0092] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0093] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the scope of the claims of the present invention.

Claims

1. A method for cleaning hydrological data of a hydrological station based on big data analysis, characterized in that, Including the following steps: Obtain the hydrological flow data at each time of the basin location collected by the hydrological station, and at the same time obtain the environment of the collection point and the operation data of the collection equipment; Based on the change of the hydrological flow data at the basin location corresponding to the time and the change of the upstream and downstream water level data, conduct an error judgment on the hydrological flow data; Conduct an abnormal assessment of the hydrological data terminal through the obtained environment of the collection point and the operation data of the collection equipment; Including the following specific steps: Obtain the environment of the collection point at the error location and the operation data of the hydrological data terminal; Based on the environmental impact at the collection time of the collection point at the error location and the impact of the operation data of the hydrological data terminal, conduct an abnormal impact assessment of the hydrological data terminal at the collection time; Including the following specific steps: Based on the environmental data at the collection time of the collection point at the error location and the normal operation data of the hydrological data terminal, conduct a comparative analysis to obtain the environmental abnormal impact result; Based on the historical operation data of the hydrological data terminal, perform an impact analysis of the operation data of the hydrological data terminal. Among them, the calculation formula for the impact analysis of the operation data of the hydrological data terminal is: , where tc is the operation duration of the hydrological data terminal, tm is the designed life of the hydrological data terminal, Tz is the data acquisition frequency of the hydrological data terminal, Tc is the maximum value of the data acquisition frequency safety range of the hydrological data terminal, exp() is the exponential power of the natural constant e, m is the number of data acquisitions of the hydrological data terminal, zj is the acquisition data of the jth acquisition, z(j - 1) is the acquisition data of the (j - 1)th acquisition, and zs is the range value of the range of the hydrological data terminal; Obtain the environmental abnormal impact result and the analysis result of the impact of the operation data of the hydrological data terminal, and perform weighted summation to obtain the abnormal impact assessment result of the hydrological data terminal; Obtain the error judgment result of the hydrological flow data and the abnormal assessment result of the hydrological data terminal to conduct a judgment on the authenticity of the hydrological flow data; According to the judgment result of the authenticity of the hydrological flow data, conduct cleaning on the hydrological flow data to obtain the cleaned hydrological flow data.

2. The method for cleaning hydrological data of a hydrological station based on big data analysis according to claim 1, wherein, The error judgment of the hydrological flow data based on the change of the hydrological flow data at the basin location corresponding to the time and the change of the upstream and downstream water level data includes the following specific steps: Obtain the hydrological flow data at the monitoring location at each monitoring time point within the corresponding period, and at the same time obtain the hydrological flow data at the upstream and downstream monitoring locations corresponding to the monitoring location, and at the same time obtain the weather environment data within the monitoring time interval; Based on the hydrological flow data at the upstream and downstream monitoring locations corresponding to the monitoring location in the historical period, the weather environment within the time interval of the upstream and downstream monitoring locations corresponding to the historical monitoring location, and the hydrological flow data at the corresponding monitoring location in the historical data period, construct a neural network model to estimate the predicted value of the hydrological flow data at the monitoring location at each monitoring time point within the corresponding period; Obtain the predicted value of the hydrological flow data at the monitoring location at each monitoring time point within the corresponding period and the actual value of the hydrological flow data at the monitoring location at each monitoring time point within the corresponding period, and import the actual value and the predicted value of the hydrological flow data at the monitoring location at the corresponding time point into the error judgment value calculation formula to calculate the error judgment value; Compare the calculated error judgment value of the monitoring location at the corresponding time point with the set error judgment threshold. If the error judgment value of the monitoring location is greater than or equal to the set error judgment threshold, set the corresponding monitoring location as the error location. If the error judgment value of the monitoring location is less than the set error judgment threshold, judge that the data is normal and no cleaning is required.

3. The method for cleaning hydrological data of a hydrological station based on big data analysis according to claim 1, wherein The judgment on the authenticity of the hydrological flow data by obtaining the error judgment result of the hydrological flow data and the abnormal assessment result of the hydrological data terminal includes the following specific steps: Obtain the error judgment value of the corresponding hydrological flow data and the abnormal impact assessment result of the hydrological data terminal at the time of the corresponding hydrological flow data; An outlier value of the hydrological flow data is obtained by weighted summation after normalizing the error judgment value of the corresponding hydrological flow data and the abnormal influence evaluation result of the hydrological data terminal respectively, and the authenticity result of the hydrological flow data is obtained by taking the reciprocal of the outlier value of the hydrological flow data.

4. The method for cleaning hydrological data of a hydrological station based on big data analysis according to claim 3, characterized in that, Cleaning the hydrological flow data according to the authenticity judgment result of the hydrological flow data to obtain the cleaned hydrological flow data, which includes the following specific contents: Compare the obtained authenticity result of the corresponding hydrological flow data with the set authenticity threshold of the hydrological flow data. If the obtained authenticity result of the corresponding hydrological flow data is greater than or equal to the set authenticity threshold of the hydrological flow data, it means that the corresponding hydrological flow data is true and does not need to be cleaned; if the obtained authenticity result of the corresponding hydrological flow data is less than the set authenticity threshold of the hydrological flow data, it means that the corresponding hydrological flow data is not true and needs to be cleaned; perform cleaning on the hydrological flow data to obtain the cleaned hydrological flow data.

5. The method for cleaning hydrological data of a hydrological station based on big data analysis according to claim 4, wherein, The steps of obtaining the hydrological data at each time of the basin location collected by the hydrological station and simultaneously obtaining the environment of the collection point and the operation data of the collection equipment include the following specific steps: The hydrological data terminals installed at each monitoring point in the basin collect the hydrological data of each monitoring point, construct the change curve of the hydrological flow data of each monitoring point over time, and store the obtained change curves of each monitoring point in the corresponding storage module. Obtain the weather environment data and electromagnetic field environment data of the monitoring location corresponding to each monitoring point and store them in the corresponding storage component. Collect the operation data of the hydrological data terminal through the operation data collection terminal. Among them, the operation data includes the operation time, the frequency of collecting data, and the fluctuation of the collected data, and store them in the corresponding storage component.

6. A hydrological data cleaning system for hydrological stations based on big data analysis, which is implemented based on the method described in any one of claims 1-5, characterized in that The system includes: A data acquisition module for acquiring the hydrological flow data at each time of the basin location collected by the hydrological station and simultaneously acquiring the environment of the collection point and the operation data of the collection equipment; an error judgment module for judging the error of the hydrological flow data based on the change of the hydrological flow data at the basin location corresponding to the time and the change of the upstream and downstream water level data; an abnormal evaluation module for evaluating the abnormality of the hydrological data terminal through the acquired environment of the collection point and the operation data of the collection equipment; a data authenticity judgment module for obtaining the error judgment result of the hydrological flow data and the abnormal evaluation result of the hydrological data terminal to judge the authenticity of the hydrological flow data; a cleaning module for cleaning the hydrological flow data according to the authenticity judgment result of the hydrological flow data to obtain the cleaned hydrological flow data.

7. An electronic device, comprising: A processor and a memory, wherein the memory stores a computer program that can be called by the processor; characterized in that the processor executes the method according to any one of claims 1-5 by calling the computer program stored in the memory.

Images