Hydrological station hydrological data cleaning method and system based on big data analysis

Abstract

The invention relates to the technical field of big data analysis, in particular to a hydrological station hydrological data cleaning method and system based on big data analysis, and the method carries out the error judgment of hydrological flow data based on the change of the hydrological flow data of a drainage basin position at a corresponding time and the change condition of upstream and downstream water level data. According to the acquired environment of the acquisition point and the operation data of the acquisition equipment, the abnormality of the hydrological data terminal is evaluated, and the error judgment result of the hydrological flow data and the abnormality evaluation result of the hydrological data terminal are acquired to judge the authenticity of the hydrological flow data. Cleaning the hydrological flow data according to the authenticity judgment result of the hydrological flow data to obtain the cleaned hydrological flow data, and accurately judging the authenticity of the obtained hydrological flow data based on the hydrological data error and the comprehensive analysis of the abnormal evaluation of the hydrological data terminal. And the accuracy and safety of data cleaning are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data analysis, and in particular to a method and system for cleaning hydrological data of a hydrological station based on big data analysis. Background Art

[0002] A large amount of hydrological data will be generated when hydrological flow monitoring is carried out at a hydrological station. However, these hydrological data contain a large number of erroneous data such as fluctuations or mutations that do not conform to reality due to errors in hydrological equipment or equipment failures. At this time, the hydrological data of the hydrological station needs to be cleaned. The existing algorithm calculates the water level slope and difference based on the water level data and time of two adjacent times. The algorithm makes judgments on the inner and outer layers. The outer layer is initially filtered and the inner layer is advanced filtered. The outer layer is strictly different from the slope and the inner layer is strictly different from the difference. The purpose is to filter out abnormal data with large mutations and high slopes. The filtering method with slope as the main and difference as the auxiliary is used to ensure the accuracy of abnormal data processing and avoid misprocessing of normal data. However, it is impossible to filter out abnormal data based on hydrological data errors and hydrological data. The comprehensive analysis of the abnormal assessment of the terminal is not accurate in judging the authenticity of the obtained hydrological flow data; the prior art, for example, discloses a hydrological data cleaning method and system in a Chinese patent with authorization announcement number CN113377750B, to obtain the hydrological data to be processed; the hydrological data to be processed is cleaned for the first time; the first cleaning includes: missing data monitoring and unreasonable data removal and filling; the hydrological data after the first cleaning is cleaned for the second time; the second cleaning includes: removal and filling of data with inconsistent causal relationships; and the cleaning results are tested. The multivariate secondary cleaning of hydrological data can provide reasonable and consistent data for further hydrological research. The prior art has the technical problems raised in this application.

[0003] In order to solve the technical problems raised in this application, this application designs a hydrological data cleaning method and system for hydrological stations based on big data analysis. Summary of the invention

[0004] In order to overcome the defects and shortcomings of the prior art, the present invention provides a hydrological data cleaning method and system for a hydrological station based on big data analysis.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a method for cleaning hydrological data of a hydrological station based on big data analysis, comprising the following steps: S1. Obtain the hydrological flow data of the basin location at each time collected by the hydrological station, and at the same time obtain the environment of the collection point and the operation data of the collection equipment; S2, judging the error of the hydrological flow data based on the change of the hydrological flow data of the basin location at the corresponding time and the change of the upstream and downstream water level data; S3, perform abnormal evaluation of the hydrological data terminal by obtaining the environment of the collection point and the operation data of the collection equipment; S4, obtaining the error judgment result of the hydrological flow data and the abnormal evaluation result of the hydrological data terminal to judge the authenticity of the hydrological flow data; S5. Clean the hydrological flow data according to the authenticity judgment result of the hydrological flow data to obtain cleaned hydrological flow data.

[0006] In one implementation of the present invention, the step of obtaining the hydrological data of the basin location at each time collected by the hydrological station and simultaneously obtaining the environment of the collection point and the operation data of the collection equipment includes the following specific steps: S101, a hydrological data terminal installed at each monitoring point in the river basin collects hydrological data at each monitoring point, constructs a curve of the hydrological flow data of each monitoring point changing with time, and stores the acquired change curve of each monitoring point in a corresponding storage module; S102, obtaining weather environment data and electromagnetic field environment data of the monitoring location corresponding to each monitoring point, and storing them in the corresponding storage component, because the weather environment and electromagnetic field environment of the monitoring location will have a negative impact on the accuracy of the acquisition terminal; S103, collecting operation data of the hydrological data terminal through the operation data collection terminal, wherein the operation data includes operation time, frequency of data collection and fluctuation of data collection, and is stored in a corresponding storage component, so as to comprehensively analyze the aging probability of the hydrological data terminal.

[0007] In one implementation of the present invention, the error judgment of the hydrological flow data is performed based on the change of the hydrological flow data of the basin location at the corresponding time and the change of the upstream and downstream water level data in step S2, including the following specific steps: S201, obtaining the hydrological flow data of the monitoring location at each monitoring time point in the corresponding period, and simultaneously obtaining the hydrological flow data of the upstream and downstream monitoring locations of the corresponding monitoring location, and simultaneously obtaining the weather environment data within the monitoring time interval; S202, constructing a neural network model to estimate the estimated value of the hydrological flow data of the monitoring location at each monitoring time point in the corresponding period based on the hydrological flow data of the upstream and downstream monitoring locations of the corresponding monitoring location in the historical period, the weather environment in the time interval of the upstream and downstream monitoring locations of the corresponding monitoring location in the historical data period, and the hydrological flow data of the corresponding monitoring location in the historical data period; S203, obtaining the estimated value of the hydrological flow data at the monitoring location at each monitoring time point in the corresponding period and the actual value of the hydrological flow data at the monitoring location at each monitoring time point in the corresponding period, and importing the actual value and the estimated value of the hydrological flow data at the monitoring location at the corresponding time point into the error judgment value calculation formula to calculate the error judgment value. In a very short period of time, the flow fluctuation is very small. If there is a large fluctuation, it means that the inaccuracy of the equipment measurement is large; S204. Compare the calculated monitoring position error judgment value of the corresponding time point with the set error judgment threshold. If the monitoring position error judgment value is greater than or equal to the set error judgment threshold, set the corresponding monitoring position as the error position and proceed to S3. If the monitoring position error judgment value is less than the set error judgment threshold, the data is judged to be normal and does not need to be cleaned.

[0008] In one implementation of the present invention, in step S3, the abnormality assessment of the hydrological flow data is performed by acquiring the environment of the collection point and the operation data of the collection equipment, including the following specific steps: S301, obtaining the environment of the error location collection point and the operation data of the hydrological data terminal; S302, evaluating the abnormal impact of the hydrological data terminal at the time of collection based on the environmental impact of the error position collection point at the time of collection and the operational data impact of the hydrological data terminal at the time of collection; Among them, the abnormal impact assessment of hydrological data terminals includes the following specific steps: S3021, comparing and analyzing the environmental data at the collection time of the error position collection point with the normal operation data of the hydrological data terminal to obtain the result of the environmental abnormality impact; S3022. Performing an impact analysis on the operation data of the hydrological data terminal based on the historical operation data of the hydrological data terminal, since the fluctuation of the collected data will affect the stability of the equipment, and frequent fluctuations in data collection will cause equipment wear; S3023. Obtain the environmental anomaly impact results and the hydrological data terminal operation data impact analysis results, perform weighted summation, and obtain the hydrological data terminal anomaly impact assessment results.

[0009] In one implementation of the present invention, obtaining the error judgment result of the hydrological flow data and the abnormal evaluation result of the hydrological data terminal in step S4 to judge the authenticity of the hydrological flow data includes the following specific steps: S401, obtaining an error judgment value corresponding to the hydrological flow data and an abnormal impact assessment result of the hydrological data terminal corresponding to the hydrological flow data moment; S402, the error judgment value of the corresponding hydrological flow data and the abnormal impact assessment result of the hydrological data terminal are standardized and weightedly summed to obtain the abnormal value of the hydrological flow data, and the authenticity result of the hydrological flow data is obtained by taking the inverse of the abnormal value of the hydrological flow data.

[0010] In one implementation of the present invention, in step S5, the hydrological flow data is cleaned according to the authenticity judgment result of the hydrological flow data to obtain the cleaned hydrological flow data, including the following specific contents: The authenticity result of the corresponding hydrological flow data obtained is compared with the set authenticity threshold of the hydrological flow data. If the authenticity result of the corresponding hydrological flow data obtained is greater than or equal to the set authenticity threshold of the hydrological flow data, it means that the corresponding hydrological flow data is authentic and does not need to be cleaned; if the authenticity result of the corresponding hydrological flow data obtained is less than the set authenticity threshold of the hydrological flow data, it means that the corresponding hydrological flow data is not authentic and needs to be cleaned; the hydrological flow data is cleaned to obtain the cleaned hydrological flow data.

[0011] In a second aspect, the present invention also provides a hydrological data cleaning system for a hydrological station based on big data analysis, specifically comprising: The data acquisition module is used to obtain the hydrological flow data at each time in the basin location collected by the hydrological station, and at the same time obtain the environment of the collection point and the operation data of the collection equipment; the error judgment module performs error judgment on the hydrological flow data based on the changes in the hydrological flow data at the basin location at the corresponding time and the changes in the upstream and downstream water level data; the anomaly assessment module is used to perform anomaly assessment on the hydrological data terminal through the environment of the collection point and the operation data of the collection equipment; the data authenticity judgment module performs nursing process anomaly analysis based on the skin vulnerability analysis results and the nursing process damage assessment results; the cleaning module cleans the hydrological flow data according to the authenticity judgment results of the hydrological flow data to obtain the cleaned hydrological flow data.

[0012] In a third aspect, the present invention provides an electronic device comprising: a processor and a memory, wherein the memory stores a computer program that can be called by the processor, and the processor executes a hydrological data cleaning method of a hydrological station based on big data analysis by calling the computer program stored in the memory.

[0013] In a fourth aspect, the present invention provides a computer-readable storage medium storing instructions, which, when executed on a computer, enables the computer to execute a method for cleaning hydrological data of a hydrological station based on big data analysis.

[0014] Compared with the prior art, the present invention has the following advantages and beneficial effects: The present invention makes an error judgment on the hydrological flow data based on the change of the hydrological flow data at the basin location at the corresponding time and the change of the upstream and downstream water level data, makes an abnormality assessment on the hydrological data terminal by acquiring the environment of the collection point and the operation data of the collection equipment, obtains the error judgment result of the hydrological flow data and the abnormality assessment result of the hydrological data terminal to make an authenticity judgment on the hydrological flow data, cleans the hydrological flow data according to the authenticity judgment result of the hydrological flow data to obtain the cleaned hydrological flow data, and makes an accurate judgment on the authenticity of the acquired hydrological flow data based on a comprehensive analysis of the hydrological data error and the abnormality assessment of the hydrological data terminal, thereby improving the accuracy and security of data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 It is a schematic diagram of the overall process of an embodiment of the method of the present invention; Figure 2 A flowchart of error judgment in an embodiment of the method of the present invention; Figure 3 It is a flow chart of the abnormal impact assessment of the hydrological data terminal according to the method embodiment of the present invention; Figure 4 It is a schematic diagram of the structure of an embodiment of the system of the present invention. DETAILED DESCRIPTION

[0016] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0017] Example 1 Embodiment 1 provides a method embodiment regarding the technical solution.

[0018] like Figures 1 to 3 As shown, this embodiment provides a method for cleaning hydrological data of a hydrological station based on big data analysis, which specifically includes the following steps: S1. Obtain the hydrological flow data of the basin location at each time collected by the hydrological station, and at the same time obtain the environment of the collection point and the operation data of the collection equipment; In this embodiment, step S1 includes the following specific steps: S101, a hydrological data terminal installed at each monitoring point in the river basin collects hydrological data at each monitoring point, constructs a curve of the hydrological flow data of each monitoring point changing with time, and stores the acquired change curve of each monitoring point in a corresponding storage module; S102, obtaining weather environment data and electromagnetic field environment data of the monitoring location corresponding to each monitoring point, and storing them in the corresponding storage component, because the weather environment and electromagnetic field environment of the monitoring location will have a negative impact on the accuracy of the acquisition terminal; S103, collecting the operation data of the hydrological data terminal through the operation data collection terminal, wherein the operation data includes the operation time, the frequency of data collection and the fluctuation of the data collection, and storing it in the corresponding storage component, so as to comprehensively analyze the aging probability of the hydrological data terminal S2, judging the error of the hydrological flow data based on the change of the hydrological flow data of the basin location at the corresponding time and the change of the upstream and downstream water level data; In step S2, the error judgment of the hydrological flow data is performed based on the change of the hydrological flow data of the basin location at the corresponding time and the change of the upstream and downstream water level data, including the following specific steps: S201, obtaining the hydrological flow data of the monitoring location at each monitoring time point in the corresponding period, and simultaneously obtaining the hydrological flow data of the upstream and downstream monitoring locations of the corresponding monitoring location, and simultaneously obtaining the weather environment data within the monitoring time interval; S202, based on the hydrological flow data of the upstream and downstream monitoring positions of the corresponding monitoring position in the historical period, the weather environment within the time interval of the upstream and downstream monitoring positions of the corresponding monitoring position in the historical data period, and the hydrological flow data of the corresponding monitoring position in the historical data period, a neural network model is constructed to estimate the estimated value of the hydrological flow data of the monitoring position at each monitoring time point in the corresponding period, wherein the neural network model construction process includes the following specific steps: collecting historical hydrological flow data of each monitoring point (upstream, downstream, target point), and obtaining upstream and downstream weather environment data corresponding to the historical data period; dividing into a training set (such as the first 80%), a validation set (10%) and a test set (10%) in chronological order; selecting a suitable neural network according to the characteristics and complexity of the data; A network model, such as LSTM, combines the historical flow of the target point and the upstream and downstream points, plus weather data, to form an input feature vector, and the output is the estimated flow value at each time point in the future. The Adam optimizer and the mean square error loss function are selected; the model training is performed, and the early stopping method is used to prevent overfitting; the model parameters, such as learning rate, number of hidden layers, batch size, etc., are optimized using grid search or random search; the model performance is evaluated on the validation set and test set using indicators such as MAE, RMSE, and R2; cross-validation of time series segmentation is performed to ensure the stability and generalization ability of the model; the prediction results of different models are compared, and the model with the best performance is selected. The specific content of the construction method of the neural network model is a conventional technology of those skilled in the art, and will not be described in detail here; S203, obtaining the estimated value of the hydrological flow data at the monitoring location at each monitoring time point in the corresponding period and the actual value of the hydrological flow data at the monitoring location at each monitoring time point in the corresponding period, and importing the actual value and the estimated value of the hydrological flow data at the monitoring location at the corresponding time point into the error judgment value calculation formula to calculate the error judgment value, wherein the error judgment value calculation formula for the monitoring location at the corresponding time point is: , where xt is the actual value of the hydrological flow data at the monitoring location at the corresponding time point, xct is the estimated value of the hydrological flow data at the monitoring location at the corresponding time point, x(t-1) is the actual value of the hydrological flow data at the previous monitoring location, and xm is the flow fluctuation safety value. is the error proportion weight, is the weight of fluctuation proportion. In this formula, the first formula is the error judgment between the actual value and the estimated value, and the second formula is the error judgment of data fluctuation. The flow fluctuation is very small in a very short period of time. If there is a large fluctuation, it means that the inaccuracy of the equipment measurement is large. S204, comparing the calculated monitoring position error judgment value of the corresponding time point with the set error judgment threshold, if the monitoring position error judgment value is greater than or equal to the set error judgment threshold, the corresponding monitoring position is set as the error position, and S3 is performed, if the monitoring position error judgment value is less than the set error judgment threshold, it is judged that the data is normal and does not need to be cleaned; S3, perform abnormal evaluation of the hydrological data terminal by obtaining the environment of the collection point and the operation data of the collection equipment; In this embodiment, in step S3, the abnormality assessment of the hydrological flow data is performed by acquiring the environment of the collection point and the operation data of the collection equipment, including the following specific steps: S301, obtaining the environment of the error location collection point and the operation data of the hydrological data terminal; S302, evaluating the abnormal impact of the hydrological data terminal at the time of collection based on the environmental impact of the error position collection point at the time of collection and the operational data impact of the hydrological data terminal at the time of collection; Among them, the abnormal impact assessment of hydrological data terminals includes the following specific steps: S3021. Compare and analyze the environmental data at the time of collection based on the error position collection point and the normal operation data of the hydrological data terminal to obtain the environmental anomaly impact result, wherein the calculation formula of the environmental anomaly impact result is: , where N is the type of environmental impact, ci is the influencing factor of the i-th environmental impact, where the environmental impact is the type of environment that affects the acquisition accuracy of the hydrological data terminal, such as ambient temperature, humidity, etc., kit is the specific value of the i-th environment at the corresponding acquisition time, and kim is the median value of the operating safety range of the i-th environment; S3022. Perform an operation data impact analysis of the hydrological data terminal based on the historical operation data of the hydrological data terminal, wherein the calculation formula for the operation data impact analysis of the hydrological data terminal is: , where tc is the operating time of the hydrological data terminal, tm is the design life of the hydrological data terminal, Tz is the data collection frequency of the hydrological data terminal, Tc is the maximum value of the data collection frequency safety range of the hydrological data terminal, exp() is the power of the natural constant e, m is the number of data collection times of the hydrological data terminal, zj is the collection data collected for the jth time, z(j-1) is the collection data collected for the j-1th time, and zs is the range value of the hydrological data terminal, that is, the maximum value of the range minus the minimum value. Since the fluctuation of the collection data will affect the stability of the equipment, frequent data collection fluctuations will cause equipment wear; S3023, obtaining the environmental abnormality impact result and the operation data impact analysis result of the hydrological data terminal, performing weighted summation to obtain the hydrological data terminal abnormality impact assessment result; S4, obtaining the error judgment result of the hydrological flow data and the abnormal evaluation result of the hydrological data terminal to judge the authenticity of the hydrological flow data; In this embodiment, obtaining the error judgment result of the hydrological flow data and the abnormality assessment result of the hydrological data terminal in step S4 to judge the authenticity of the hydrological flow data includes the following specific steps: S401, obtaining an error judgment value corresponding to the hydrological flow data and an abnormal impact assessment result of the hydrological data terminal corresponding to the hydrological flow data moment; S402, the error judgment value of the corresponding hydrological flow data and the abnormal impact assessment result of the hydrological data terminal are standardized and weighted and summed to obtain the abnormal value of the hydrological flow data, and the abnormal value of the hydrological flow data is inversely calculated to obtain the authenticity result of the hydrological flow data, wherein the calculation result of the authenticity result of the hydrological flow data can be: , where Wc is the abnormal impact assessment result of the hydrological data terminal, is the weight of error judgment value proportion, The weight of the terminal abnormality impact assessment result; S5. Cleaning the hydrological flow data according to the authenticity judgment result of the hydrological flow data to obtain cleaned hydrological flow data; In this embodiment, in step S5, the hydrological flow data is cleaned according to the authenticity judgment result of the hydrological flow data to obtain the cleaned hydrological flow data, including the following specific contents: The authenticity result of the corresponding hydrological flow data obtained is compared with the set authenticity threshold of the hydrological flow data. If the authenticity result of the corresponding hydrological flow data obtained is greater than or equal to the set authenticity threshold of the hydrological flow data, it means that the corresponding hydrological flow data is authentic and does not need to be cleaned; if the authenticity result of the corresponding hydrological flow data obtained is less than the set authenticity threshold of the hydrological flow data, it means that the corresponding hydrological flow data is not authentic and needs to be cleaned; the hydrological flow data is cleaned to obtain the cleaned hydrological flow data.

[0019] It should be noted in this embodiment that the setting parameters in this embodiment (such as various weights and proportions, thresholds, etc.) are obtained by experiments by technicians in this field. The specific experimental method is: obtain hydrological flow data at different times in the basin location collected by several groups of historical hydrological stations, and at the same time obtain the historical environment of the collection point and the operation data of the collection equipment, substitute them into the steps of this embodiment to calculate the authenticity results of the corresponding hydrological flow data, and at the same time obtain the judgment results of the field detection of whether the hydrological flow is real. Based on the authenticity results of the hydrological flow data and the judgment results of the field detection of whether the hydrological flow is real, the fitting software (preferably MATLAB) is imported to perform iterative fitting of the data, and the setting parameter values ​​that meet the maximum judgment accuracy are output.

[0020] It should be noted in this embodiment that this embodiment has the following benefits: error judgment of the hydrological flow data is performed based on changes in the hydrological flow data of the basin location at the corresponding time and changes in the upstream and downstream water level data; abnormality assessment of the hydrological data terminal is performed by obtaining the environment of the collection point and the operation data of the collection equipment; the error judgment result of the hydrological flow data and the abnormal assessment result of the hydrological data terminal are obtained to judge the authenticity of the hydrological flow data; the hydrological flow data is cleaned according to the authenticity judgment result of the hydrological flow data to obtain the cleaned hydrological flow data; based on the comprehensive analysis of the hydrological data error and the abnormal assessment of the hydrological data terminal, the authenticity of the acquired hydrological flow data is accurately judged, thereby improving the accuracy and security of data cleaning.

[0021] Example 2 like Figure 4 As shown, this embodiment provides a hydrological data cleaning system for a hydrological station based on big data analysis, including: The data acquisition module is used to obtain the hydrological flow data of the basin location at each time collected by the hydrological station, and at the same time obtain the environment of the collection point and the operation data of the collection equipment; the error judgment module is used to make an error judgment on the hydrological flow data based on the changes in the hydrological flow data of the basin location at the corresponding time and the changes in the upstream and downstream water level data; the abnormality assessment module is used to make an abnormality assessment of the hydrological data terminal through the environment of the collection point and the operation data of the collection equipment; the data authenticity judgment module is used to perform an abnormality analysis of the nursing process based on the skin vulnerability analysis results and the nursing process damage assessment results; the cleaning module is used to clean the hydrological flow data according to the authenticity judgment results of the hydrological flow data to obtain the cleaned hydrological flow data. The above-mentioned parameters in the hydrological data cleaning system of the hydrological station based on big data analysis of the present invention and the steps and corresponding effects of each unit module to realize the corresponding functions can refer to the parameters and steps in the embodiment of the hydrological data cleaning method of the hydrological station based on big data analysis in the above text, and will not be repeated here.

[0022] Example 3 An electronic device according to an embodiment of the present invention includes: a processor and a memory, wherein the memory stores a computer program that can be called by the processor, and the processor executes a hydrological data cleaning method for a hydrological station based on big data analysis by calling the computer program stored in the memory. It should be noted that all computer programs of the hydrological data cleaning method for a hydrological station based on big data analysis are implemented using C language.

[0023] Example 4 This embodiment provides a computer-readable storage medium having a rewritable computer program stored thereon; When the computer program runs on a computer device, the computer device executes the above-mentioned hydrological station hydrological data cleaning method based on big data analysis.

[0024] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the IoT device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0025] The system and medium provided in the embodiments of the present invention correspond one-to-one to the method, and therefore, the system and medium also have similar beneficial technical effects to the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the system and medium will not be repeated here.

[0026] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0027] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0028] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0029] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0030] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0031] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0032] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0033] The above are only embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A hydrological data cleaning method for a hydrological station based on big data analysis, characterized in that: The steps include: Obtain the hydrological flow data of the basin location at each time collected by the hydrological station, and at the same time obtain the environment of the collection point and the operation data of the collection equipment; The error judgment of hydrological flow data is performed based on the changes in hydrological flow data at the basin location at the corresponding time and the changes in upstream and downstream water level data; Abnormal evaluation of hydrological data terminals is performed by obtaining the environment of the collection point and the operation data of the collection equipment; Obtain the error judgment results of the hydrological flow data and the abnormal evaluation results of the hydrological data terminal to judge the authenticity of the hydrological flow data; The hydrological flow data is cleaned according to the authenticity judgment result of the hydrological flow data to obtain the cleaned hydrological flow data.

2. The method for cleaning hydrological data of a hydrological station based on big data analysis according to claim 1 is characterized in that: The error judgment of the hydrological flow data based on the change of the hydrological flow data of the basin location at the corresponding time and the change of the upstream and downstream water level data includes the following specific steps: Obtain the hydrological flow data of the monitoring location at each monitoring time point within the corresponding period, and simultaneously obtain the hydrological flow data of the upstream and downstream monitoring locations of the corresponding monitoring location, and simultaneously obtain the weather environment data within the monitoring time interval; Based on the hydrological flow data of the upstream and downstream monitoring positions of the corresponding monitoring position in the historical period, the weather environment in the time interval of the upstream and downstream monitoring positions of the corresponding monitoring position in the historical data period, and the hydrological flow data of the corresponding monitoring position in the historical data period, a neural network model is constructed to estimate the estimated value of the hydrological flow data of the monitoring position at each monitoring time point in the corresponding period; Obtain the estimated value of the hydrological flow data at the monitoring location at each monitoring time point in the corresponding period and the actual value of the hydrological flow data at the monitoring location at each monitoring time point in the corresponding period, and import the actual value and the estimated value of the hydrological flow data at the monitoring location at the corresponding time point into the error judgment value calculation formula to calculate the error judgment value; The calculated monitoring position error judgment value of the corresponding time point is compared with the set error judgment threshold. If the monitoring position error judgment value is greater than or equal to the set error judgment threshold, the corresponding monitoring position is set as the error position. If the monitoring position error judgment value is less than the set error judgment threshold, the data is judged to be normal and does not need to be cleaned.

3. The method for cleaning hydrological data of a hydrological station based on big data analysis according to claim 1 is characterized in that: The abnormal evaluation of hydrological flow data by obtaining the environment of the collection point and the operation data of the collection equipment includes the following specific steps: Obtain the environment of the error location collection point and the operating data of the hydrological data terminal; The abnormal impact of the hydrological data terminal at the time of collection is evaluated based on the environmental impact of the error position collection point at the time of collection and the operational data impact of the hydrological data terminal.

4. The method for cleaning hydrological data of a hydrological station based on big data analysis according to claim 3 is characterized in that: The hydrological data terminal abnormal impact assessment includes the following specific steps: The environmental data at the time of collection based on the error position collection point is compared with the normal operation data of the hydrological data terminal to obtain the results of environmental abnormality impact; The operation data impact analysis of the hydrological data terminal is performed based on the historical operation data of the hydrological data terminal. The calculation formula for the operation data impact analysis of the hydrological data terminal is: , where tc is the operating time of the hydrological data terminal, tm is the design life of the hydrological data terminal, Tz is the data collection frequency of the hydrological data terminal, Tc is the maximum value of the data collection frequency safety range of the hydrological data terminal, exp() is the power of the natural constant e, m is the number of data collection times of the hydrological data terminal, zj is the data collected for the jth time, z(j-1) is the data collected for the j-1th time, and zs is the range value of the range of the hydrological data terminal; The abnormal environmental impact results and the operational data impact analysis results of the hydrological data terminal are obtained by performing weighted summation to obtain the abnormal impact assessment results of the hydrological data terminal.

5. The method for cleaning hydrological data of a hydrological station based on big data analysis according to claim 1 is characterized in that: The step of obtaining the error judgment result of the hydrological flow data and the abnormal evaluation result of the hydrological data terminal to judge the authenticity of the hydrological flow data includes the following specific steps: Obtain the error judgment value corresponding to the hydrological flow data and the abnormal impact assessment result of the hydrological data terminal corresponding to the hydrological flow data moment; The abnormal value of the hydrological flow data is obtained by standardizing the error judgment value of the corresponding hydrological flow data and the abnormal impact assessment result of the hydrological data terminal and then taking the weighted sum. The authenticity result of the hydrological flow data is obtained by taking the inverse of the abnormal value of the hydrological flow data.

6. The method for cleaning hydrological data of a hydrological station based on big data analysis according to claim 5 is characterized in that: The cleaning of the hydrological flow data according to the authenticity judgment result of the hydrological flow data to obtain the cleaned hydrological flow data includes the following specific contents: The authenticity result of the corresponding hydrological flow data obtained is compared with the set authenticity threshold of the hydrological flow data. If the authenticity result of the corresponding hydrological flow data obtained is greater than or equal to the set authenticity threshold of the hydrological flow data, it means that the corresponding hydrological flow data is authentic and does not need to be cleaned; if the authenticity result of the corresponding hydrological flow data obtained is less than the set authenticity threshold of the hydrological flow data, it means that the corresponding hydrological flow data is not authentic and needs to be cleaned; the hydrological flow data is cleaned to obtain the cleaned hydrological flow data.

7. The method for cleaning hydrological data of a hydrological station based on big data analysis according to claim 6 is characterized in that: The method of obtaining the hydrological data of the basin location at each time collected by the hydrological station and simultaneously obtaining the environment of the collection point and the operation data of the collection equipment includes the following specific steps: The hydrological data terminal installed at each monitoring point in the basin collects the hydrological data of each monitoring point, constructs the change curve of the hydrological flow data of each monitoring point over time, and stores the obtained change curve of each monitoring point in the corresponding storage module; Acquire weather environment data and electromagnetic field environment data corresponding to the monitoring location of each monitoring point, and store them in the corresponding storage component; The operation data of the hydrological data terminal is collected through the operation data collection terminal, wherein the operation data includes the operation time, the frequency of collecting data and the fluctuation of the collected data, and is stored in the corresponding storage component.

8. A hydrological data cleaning system for a hydrological station based on big data analysis, which is implemented based on the method described in any one of claims 1 to 7, characterized in that: The system comprises: The data acquisition module is used to obtain the hydrological flow data at each time in the basin location collected by the hydrological station, and at the same time obtain the environment of the collection point and the operation data of the collection equipment; the error judgment module performs error judgment on the hydrological flow data based on the changes in the hydrological flow data at the basin location at the corresponding time and the changes in the upstream and downstream water level data; the anomaly assessment module is used to perform anomaly assessment on the hydrological data terminal through the environment of the collection point and the operation data of the collection equipment; the data authenticity judgment module performs nursing process anomaly analysis based on the skin vulnerability analysis results and the nursing process damage assessment results; the cleaning module cleans the hydrological flow data according to the authenticity judgment results of the hydrological flow data to obtain the cleaned hydrological flow data.

9. An electronic device, comprising: A processor and a memory, wherein the memory stores a computer program that can be called by the processor; characterized in that the processor executes the method according to any one of claims 1 to 7 by calling the computer program stored in the memory.

Images