Hydrological data grading collaborative treatment method

Abstract

The invention discloses a hydrological data grading collaborative treatment method, and aims to solve the problems of hydrological multi-source data fusion distortion and poor cross-node consistency. The method comprises the following steps: establishing a mapping relation library, and converging real-time data of each service system device through double channels; executing automatic switching of the main and standby equipment based on the physical range and the priority, and fusing to generate main data of the observation station; monitoring the unblocked rate and the water level difference rate of the equipment in a full-link manner, obtaining the physical objective state of the equipment, triggering a linkage mechanism when a fault or an overrange occurs, automatically removing abnormal equipment, calculating a cardinal number, and blocking data warehousing; calculating a false alarm rate and a missing report rate based on a user manual correction record, and dynamically optimizing and updating a quality control rule triggering threshold value; and packaging and synchronizing the standardized data set, writing the data set into a local cache queue when a network fault occurs, and continuously transmitting the data set in sequence after recovery. According to the method, the robustness and the strong consistency of the underlying data subjected to stage processing are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of hydrology and relates to hydrological data processing in a multi-level architecture system, particularly to a hierarchical collaborative governance method for hydrological data. Background Technology

[0002] Hydrological data is a core foundation for flood and drought disaster prevention, water resource allocation, and the construction of digital twin watersheds. With the rapid expansion of the Internet of Things (IoT) monitoring and sensing system, the central nodes of provincial data centers aggregate massive amounts of real-time hydrological and rainfall data from multiple sources. However, against the backdrop of surging data volume and diversified applications, the existing hydrological data flow and processing architecture faces the following prominent technical shortcomings: First, the heterogeneous multi-source data fusion and transfer mechanism has defects. When processing heterogeneous data from multiple concurrent devices at the same station, the existing system lacks an underlying two-way real-time synchronization and physical linkage mechanism for device status. This results in the system being unable to block abnormal data from entering the data warehouse and perform hot switching of the primary and backup links at the underlying level based on the physical objective status of the devices (such as exceeding the range or failure), causing data distortion and poor consistency when aggregated to the data base.

[0003] Secondly, the monitoring and early warning rules are rigid and lack adaptive evolution capabilities. Traditional systems often use static, hard-coded thresholds (such as fixed water level difference limits) for data cleaning and alarms, which are difficult to adapt to the complex and ever-changing real-world hydrological environment. These rigid rules not only lead to a large number of false alarms or missed alarms in the monitoring system, but also lack a bidirectional state synchronization and rule coordination mechanism in a distributed network. Sub-nodes at all levels cannot locally optimize algorithm thresholds based on local flood control characteristics and synchronize them in reverse, and the master node also lacks the ability to receive state correction command streams from sub-nodes to perform global threshold adaptive optimization, resulting in the inability of the entire network's computing power to effectively evolve collaboratively.

[0004] Finally, the data consistency guarantee for multi-level nodes in the provincial, municipal, and county distributed architecture is insufficient. Existing data flow synchronization mainly relies on conventional transmission protocols and fails to establish a directional routing and push mechanism based on data logical identifiers at the underlying level. When encountering network shocks or interruptions caused by severe hydrological environments, there is a lack of reliable local queue caching and an in-order resumption mechanism after network recovery. This easily leads to the loss of underlying data packets, making it impossible to guarantee eventual data consistency among provincial, municipal, and county-level nodes. Summary of the Invention

[0005] To address the aforementioned deficiencies in existing technologies, this invention provides a hierarchical collaborative governance method for hydrological data. This method aims to solve technical problems such as distortion in multi-source fusion of hydrological data, discontinuity in rule collaboration among distributed nodes, rigidity of fixed monitoring thresholds, and poor data consistency among nodes at all levels by employing physical state-linked data entry blocking, logical identifier-based directional routing push, rule-based collaborative optimization based on edge command stream feedback, and a network interruption resumption mechanism based on a cache queue.

[0006] The present invention addresses the aforementioned technical problems primarily through the following technical solution: a method for hierarchical and collaborative management of hydrological data, comprising the following steps: S1: Establish a mapping relationship library of "station-equipment-element-index value" containing administrative region codes, and aggregate the real-time equipment data scattered in various business systems into a data base containing a raw data warehouse and an application data warehouse; wherein, the real-time equipment data enters the warehouse through the first channel and is stored in the raw data warehouse on a per-equipment basis, and enters the application data warehouse after cleaning; based on the mapping relationship library, the raw data at the equipment level is automatically merged into the corresponding monitoring station, and the monitoring equipment that is in the current data application state (i.e., the highest priority and within the effective measurement range) is marked as the main equipment, and the rest are marked as secondary equipment, forming a standardized dataset containing the main and secondary equipment identifiers on a per-station basis; This step achieves low-level integration from isolated data at the device level in heterogeneous systems to standard data at the station level through a mapping relationship library. By implementing dynamic determination of physical device state machines and primary / backup identification at the data inbound source, a definite logical pointer is provided for subsequent hot switching of underlying routing links.

[0007] S2: The master node monitors the entire data link of the device in real time, calculates the device smoothness rate in real time with the device as the basic unit, and calculates the water level difference rate of the master and slave devices; when the device smoothness rate or the water level difference rate is detected to be worse than the preset limit, data with abnormal event markers is generated as alarm data, and the alarm data is pushed to the corresponding level of child nodes based on the administrative code of the station code in the mapping relationship library. This step overcomes the limitations of traditional centralized web polling or global broadcasting. By parsing the logical identifier (administrative region code) in the data primary key (station code) at the network layer, point-to-point targeted routing and pushing of abnormal messages are achieved, significantly reducing the distribution pressure on the master node and the bandwidth consumption of the backbone network under high concurrency conditions.

[0008] S3: The master node uses the built-in quality control rule set to verify the aggregated data; at the same time, it acquires the physical objective status of the monitoring equipment in real time and marks it dynamically. When the monitoring equipment is marked as faulty, out of service, or in an abnormal state that exceeds the effective range of the water level equipment, the linkage control mechanism is triggered to block the real-time data of the abnormal equipment from entering the application warehouse. The fundamental purpose of this step is to establish a low-level hard isolation between physical state and data calculation. When physical equipment malfunctions (such as being blocked by foreign objects causing over-range measurement), the system not only removes it from the statistical base, but also implements physical blocking at the data channel level to prevent local defects from contaminating the overall water conservancy map.

[0009] S4: The master node receives a status correction instruction stream from each level of sub-nodes based on the pushed data feedback. The status correction instruction stream includes manual correction operations and ignore operations triggered by users at each level for alarm data with abnormal event markers. Based on the aggregated status correction instruction stream, the master node calculates the false alarm rate and false negative rate of each rule in the quality control rule set. When the false alarm rate or false negative rate exceeds a preset threshold, the trigger threshold of the corresponding rule is automatically adjusted up or down by a certain amount. This step constructs a bottom-up edge feedback closed loop. It seamlessly transforms the human interactions distributed across various edge sub-nodes into a machine-readable serialized "state correction instruction stream," which is then fed back to the central algorithm library via an asynchronous channel. This endows the distributed quality control model with generalization learning and adaptive evolution capabilities at the provincial scale. As the preferred algorithm, the system can extract the distribution of historical real-world abnormal water condition features that were manually marked as "ignored," taking the 95th percentile as a candidate threshold, supplemented by flood control feature values ​​as a safety upper limit, to achieve automatic optimization driven by computing power.

[0010] S5: The standardized dataset processed by the master node and the optimized quality control rule set are packaged into standard format messages and synchronously distributed to sub-nodes at all levels and various business applications to achieve cross-level data and rule synchronization and collaboration. During the synchronous push process, the sending status of each message is recorded. If a network failure causes the push to fail, the message is automatically written to a local cache queue. After the network is restored, the messages in the local cache queue are transmitted in order. For extreme network outages faced by hydrological flood control, this solution introduces a time-sensitive dual-channel queue splitting mechanism to ensure that the highest priority (e.g., the real-time flood peak message within the last 5 minutes) can be prioritized for distribution the moment the network is restored, while other historical messages are asynchronously and rate-limitedly replenished.

[0011] Preferably, S3 also includes a bottom-up collaborative mechanism for governance rules: each level of sub-nodes obtains the flood control characteristic values ​​of the monitoring stations in their jurisdiction and the operation status of the supporting water conservancy projects, performs local optimization of the thresholds in the quality control rule set station by station, and synchronizes the optimized rule parameters back to the provincial rule library of the master node through the vertical synchronization data stream to take effect immediately. At the same time, the master node automatically retains collaborative modification records containing the modifier, modification time and modification reason.

[0012] As a preferred embodiment, in step S1, when multiple sets of monitoring devices at the same station report data concurrently for the same element, the backup device is automatically switched to the current data application device based on the preset effective range of the water level device and the priority of the station application device. The monitoring device in the current data application state is marked as the main device, and the other devices are marked as auxiliary devices.

[0013] When automatically switching from backup equipment to master equipment, the system can also embed a dynamic data quality confidence score model at the bottom layer. This model calculates the physical range deviation of the equipment (state factor, preferred weight 0.4-0.6), variance mutation degree (fluctuation factor, preferred weight 0.2-0.4), and historical arrival rate (reputation factor, preferred weight 0.1-0.3). This weighted score serves as the arbitration basis for master data entry in the event of concurrent conflicts, thereby greatly improving the data's noise resistance.

[0014] Preferably, the formula for calculating the correlation ratio in S2 is as follows: The equipment availability rate = (number of devices that reported data in the first hour / total number of valid devices) × 100%, where the total number of valid devices refers to the number of devices associated with the station that have not been marked as out of service or faulty by the system. The water level difference rate is calculated as follows: (Number of stations with a water level difference greater than the preset limit between the main and auxiliary equipment / Total number of stations with main and auxiliary equipment) × 100%.

[0015] The key is to dynamically determine the total number of valid devices in the denominator of the connectivity rate formula. By removing “out of service” or “faulty” devices from the denominator in real time, the connectivity rate indicator can truly reflect the health of the currently available network communication and avoid false network-wide alarms caused by planned equipment maintenance.

[0016] Preferably, the formula for calculating the correlation ratio in S4 is as follows: The false positive rate = number of anomalies marked as "ignore" by the user / total number of times the rule is triggered; The false negative rate = the number of anomalies that were manually corrected by the user but not identified by the rules / the total number of manual corrections by the user.

[0017] The above formula clarifies the evaluation function for machine adaptive evolution. The false positive rate measures the oversensitivity of the system rules, while the false negative rate measures the oversensitivity. By strictly defining the sources of the numerator and denominator of these two ratios, the daily click-to-confirm / ignore operations of business personnel at the provincial, municipal, and county levels are seamlessly transformed into labeled training samples for deep learning or threshold optimization algorithms.

[0018] Preferably, the built-in quality control rule set in S3 includes the upper limit of error, the lower limit of error, the upper limit of suspicion, the lower limit of suspicion, and the amplitude limit of the observed elements.

[0019] Error limits are typically tied to insurmountable physical extremes (e.g., water levels can never be lower than the riverbed elevation or higher than the dam crest elevation); while questionable limits are tied to empirical statistical values ​​(e.g., historical high water levels). Meanwhile, the 5-minute water level fluctuation limit is primarily used to intercept short-term communication glitches or sensor jitter.

[0020] Preferably, the station codes in the mapping relationship library in S1 are composed of 8-bit strings, including 3 bits for the watershed system code, 1 bit for the station type code, 2 bits for the administrative region code, and 2 bits for the station serial number.

[0021] The station coding structure is the construction logic of the primary key of the data base. This 8-bit coding structure (watershed system + station type + administrative region + serial number) not only ensures the global uniqueness of "one station, one code", but its segmented structure also makes it easier for the underlying database engine to directly use string prefix matching to improve index retrieval efficiency when performing spatial geographic aggregation queries or hierarchical order dispatching by administrative division.

[0022] Preferably, the specific rules for acquiring and dynamically marking the physical objective status of the monitoring equipment in real time in S3 include: marking equipment with short-term faults as faulty, marking equipment that cannot be repaired in a short time or has been removed as out of service, and automatically marking water level data as over-range abnormal when it exceeds the upper and lower limits of the effective range of the water level equipment.

[0023] The technical significance of distinguishing between faults and outages lies in the fact that faults are often sudden and disorderly, requiring immediate maintenance dispatch and data recovery; while outages are orderly states requiring manual intervention (such as routine dismantling after the flood season). Once the system recognizes an outage, it will directly silence related alarms to avoid unnecessary consumption of computing resources.

[0024] Preferably, S1 further includes: directly aggregating manual flood reporting data, basic information of monitoring stations, and historical feature data from various business systems into the application warehouse through a second channel.

[0025] For flood reporting data or relatively static basic information that has been manually verified, the data quality naturally has a high degree of confidence. By using a second channel (XE channel) to bypass the original warehouse and complex cleaning rules and directly reach the application warehouse, computing power consumption is greatly reduced, while ensuring zero delay in the entry of critical flood control instructions and manually verified data into the warehouse.

[0026] Preferably, in step S5, the standardized dataset is pushed to various business applications on demand through a standardized API interface, and the call to the standardized API interface implements a dual authentication mechanism of IP and key.

[0027] Given the sensitivity of hydrological data, a dual authentication mechanism of IP whitelist binding and dynamic key signing is implemented to ensure the legality and traceability of every API interface call, and to prevent the core data base from being illegally stolen or maliciously tampered with when providing open collaborative services.

[0028] Preferably, S5 further includes: deploying channel monitoring agents on the master node and child nodes respectively, collecting in real time the connectivity status, transmission rate, backlog data volume and error rate indicators of the synchronous distribution channel, and automatically sending alarms when the channel interruption exceeds the set time, the backlog data exceeds the threshold or the error rate exceeds the standard.

[0029] By using a bypass-deployed monitoring agent to monitor the health of the underlying TCP / IP connections and the depth of the packet backlog queue in real time, the system can detect network instability and trigger an early warning mechanism before actual business-level "data loss" occurs, thereby giving the system stronger network disaster recovery resilience.

[0030] This solution can display the real-time inbound status of various data sources, monitoring indicators of the raw data warehouse and application data warehouse, and data push and distribution status based on a full-link monitoring interface. When data anomalies are detected, the system not only triggers a highlight flashing location on the interface, but also automatically queries the list of responsible persons by station code in the system backend, and pushes the alarm details (including alarm value and station name) to the responsible person's terminal via SMS gateway. At the same time, a process line graph of the station's water level changing over time can be generated synchronously on the handling interface, and flood control characteristic values ​​such as warning water level can be overlaid on this layer to assist operators in making visual and intuitive judgments.

[0031] The substantial effects of this invention are: 1. Improved robustness and consistency of underlying data: By constructing a dual-channel data base of the original warehouse and the application warehouse, and introducing a dynamic marking and linkage blocking mechanism for monitoring the physical status of the equipment (such as the upper and lower limits of the effective range), the system can automatically remove the sensor from the calculation base of the rate value and block its entry into the warehouse the moment the sensor experiences physical drift or failure, thus ensuring the absolute reliability of the collaborative governance base data from the source of physical data.

[0032] 2. The system achieves adaptive evolution and closed-loop optimization of quality control rules: It breaks through the limitations of traditional static thresholds. The system can automatically capture and record the user's manual correction and ignore operations, dynamically calculate the false alarm rate and false alarm rate of each rule, and automatically adjust the trigger threshold upward or downward based on this. This allows the computer cleaning algorithm to continuously fit the real physical environment as the running time progresses, greatly reducing the system's ineffective computing power consumption and alarm redundancy.

[0033] 3. Enhanced network disaster recovery and communication restoration capabilities under extreme conditions: Relying on local cache queues and status monitoring mechanisms, the system can achieve local lossless temporary storage of packets when encountering network failures caused by severe environments; and perform strict in-order retransmission after network recovery, completely solving the problem of packet loss during cross-level and multi-node data synchronization, and truly achieving strong consistency of underlying data with "one modification, everywhere synchronization". Attached Figure Description

[0034] Figure 1 This is a flowchart of a hydrological data hierarchical collaborative management method according to the present invention. Detailed Implementation

[0035] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0036] Example: A hierarchical collaborative governance method for hydrological data. This method is mainly applied to the data flow and governance system of multi-level hydrological agencies from the provincial to the municipal and county levels. The provincial hydrological data center deploys a master node, and the local servers or edge gateways of the municipal and county-level hydrological bureaus deploy sub-nodes at various levels. The master node and sub-nodes construct a vertically synchronized data flow channel through the network.

[0037] like Figure 1 As shown, this method specifically includes the following steps: Step S1: Multi-source data aggregation and intelligent fusion of master data The system uses the unique code of each hydrological station as the primary key to establish a mapping relationship database of "station-equipment-element-index value," recording the equipment ID, equipment priority, and index value information corresponding to various monitoring elements under each station. The station code uses an 8-digit string, consisting of a 3-digit river system code, a 1-digit station type code, a 2-digit administrative region code, and a 2-digit station serial number, ensuring that each station in the province has a unique and unchanging code. For example, the code of a water level station in the Qiantang River basin is "7011AA01," where "701" is the Qiantang River basin river system code, "1" is the water level station type code, "AA" is the administrative region code of Shangcheng District, Hangzhou City, and "01" is the station serial number. Assume that the water level monitoring element of this station corresponds to the primary equipment ID "SW001" and the backup equipment ID "SW002," with a water level index value of 2.6; the rainfall monitoring element corresponds to the equipment ID "YL015," with a rainfall index value of 2.2.

[0038] During the data aggregation phase, real-time device data scattered across various business systems is entered into the warehouse via the first channel (ME channel). It is first stored in the raw warehouse on a per-device basis, and then enters the application warehouse after quality control cleaning. Other data that does not require cleaning (such as manual flood reporting data, basic station information, historical feature data, etc.) enters the application warehouse directly via the second channel (XE channel).

[0039] Based on the "station-equipment-element-index value" mapping rule, the system automatically merges the raw equipment-level data into the corresponding stations, forming a standardized dataset with stations as units. This dataset is then stored in the application repository according to the standard format of "Real-time Rainfall and Water Information Database Table Structure and Identifiers" SL323-2011. To address the data selection issue when multiple monitoring devices at the same station report the same element concurrently, the system performs hard switching based on the preset effective range and priority of the water level equipment.

[0040] The automatic switching process based on the effective range of water level equipment and the priority of station application equipment specifically includes the following underlying state machine logic: The system presets an independent effective water level range (including physical upper and lower limits) and priority order for each monitoring device under the same station. During the real-time data entry process, the system continuously verifies the data validity of the master device. When the master device is manually or systematically marked as 'faulty' or 'out of service', or when its real-time water level reading exceeds its dedicated effective range and is dynamically marked as 'out of range abnormal' by the system, the underlying routing of the system immediately triggers a hard switchover mechanism.

[0041] During the switchover, the system first iterates through all backup devices at the station according to a preset priority order, and simultaneously verifies whether the real-time water level of each backup device is within its effective range. The system automatically selects the backup device with the highest priority and within its effective range, and switches its data source pointer to the current data application device. This hot-swap process of the primary and backup routes is completed automatically within one minute, ensuring the continuity and uninterrupted operation of the data base aggregation layer. Simultaneously, the system automatically generates a switchover record in the log database, including the switchover time, the original primary device ID, the new primary device ID, and the specific switchover reason (e.g., 'Primary device SW001 over-range'), achieving full traceability of the entire process of physical status linkage and data link takeover. In addition to automatic switching based on the effective range of water level equipment and the priority of station application equipment, this method also introduces a dynamic data quality confidence assessment model at the underlying level. Specifically, the system calculates evaluation factors in three dimensions for concurrent devices in real time: State factor C status Calculated based on the degree to which the current reading deviates from the upper and lower limits of the device's effective range; Volatility Factor C variance : Calculate the variance between the current data and the sliding average value within the historical preset time window of the device; Reputation Factor C history The system dynamically calculates the data quality confidence score Q based on the feature arrival rate and the number of manual corrections over the past 24 hours. The feature arrival rate is the ratio of the number of features with data at the top of each hour within a selected time period to the total number of features. The system then calculates the dynamic data quality confidence score Q using the following formula: Q=w1·Cstatus +w2·C variance +w3·C history ; Wherein, w1, w2, and w3 are the weights of three factors preset by the system, and satisfy w1+w2+w3=1. In order to maximize the robustness of data fusion, this invention fully considers the principle of prioritizing physical constraints and sets the value range of the weights as follows: w1∈[0.4,0.6], w2∈[0.2,0.4], w3∈[0.1,0.3].

[0042] For example, we can choose w1=0.5, w2=0.3, and w3=0.2. The technical mechanism of this weight allocation scheme is as follows: the contribution rate of monitoring equipment approaching its physical range limit (state factor) to data distortion increases exponentially, so it is given the highest weight to achieve priority error prevention at the physical level; while transient variance (fluctuation factor), as an intermediate indicator for identifying sudden floods and sensor noise, is given a moderate weight; and historical arrival rate (reputation factor), as an auxiliary correction term to mitigate the long-term sub-health state of equipment, is given the lowest weight. Through this specific weight allocation, the system can accurately isolate distorted data from high-risk equipment under complex conditions such as sudden flood peaks.

[0043] When the data difference exceeds the safety range, the system automatically selects the data of the device with the highest score Q as the current master data and puts that device into the warehouse, and temporarily marks it as the master device, while the others are marked as slave devices, thereby achieving highly robust intelligent fusion.

[0044] The system monitors data change events in various business systems. For example, when the basic information of a test station in a business system is modified, the system immediately captures the change event, triggers incremental synchronization, writes the updated information to the data base, and records the change log, including the person who made the modification, the modification time, and the modified content.

[0045] The system also periodically accesses external information such as water conservancy maps, administrative divisions, and engineering operation status through external interfaces to enrich the data correlation dimensions of the monitoring stations.

[0046] Step S2: End-to-end data monitoring and dynamic limit comparison The system monitors the entire data flow process and sets two major monitoring indicators, namely equipment smoothness rate and hydrological difference rate, with equipment as the basic unit.

[0047] The formula for calculating the equipment availability rate is: Equipment availability rate = (Number of devices that reported data in the first hour / Total number of valid devices) × 100%.

[0048] Among them, the total number of valid devices refers to the number of devices associated with the station that are not marked as "out of service" or "faulty"; it is automatically calculated and updated every hour, and an alarm is triggered when it is below 80%; the device availability rate is automatically calculated every hour: the system counts the number of devices that reported data in the previous hour, divides it by the total number of valid devices (the number of devices associated with the station that are not marked as "out of service" or "faulty"), and obtains the device availability rate.

[0049] The formula for calculating the water level difference rate is: Water level difference rate = (Number of stations with water level difference between main and auxiliary equipment > limit / Total number of stations with main and auxiliary equipment) × 100%. The water level difference between main and auxiliary equipment is calculated daily at 8:00 AM. Stations exceeding the default limit by 0.2 meters are included in the "List of Water Level Differences Between Main and Auxiliary Equipment". During the statistics, cases where any equipment exceeds its range or is marked as faulty are excluded. The daily calculation of the water level difference rate at 8:00 AM involves the system filtering stations with both main and auxiliary equipment that are not reported as out of service or faulty. The system calculates the water level difference between main and auxiliary equipment at 8:00 AM, and stations exceeding this limit by 0.2 meters are included in the "List of Water Level Differences Between Main and Auxiliary Equipment". For example, if a station's main equipment water level is 27.35m and the auxiliary equipment water level is 27.58m, the water level difference is 0.23m, exceeding the limit. The system automatically includes this station in the list and associates it with the responsible person's information.

[0050] Automatically generate a "1-hour data-free device list", which lists devices that have no data in the last hour, including device ID, station code, station name, last data time, device type, responsible person, and contact number; and calculate a "24-hour data-missing device list", which accumulates the total duration of data loss for each device in the past 24 hours and sorts them in descending order of duration of data loss. Real-time statistics of the arrival rate of all elements, including hourly data arrival status of elements such as reservoir water level, river water level, and rainfall; the formula for calculating the element arrival rate is: Total element arrival rate = (Number of elements with data at the hour within the selected time period / Total number of elements) × 100%; When the equipment accessibility rate, water level difference rate, element arrival rate, or accumulated abnormal data exceed the threshold, data marked with abnormal events is generated as alarm data. Based on the administrative region code of the station code in the mapping relationship database, the alarm data is pushed to the corresponding child node, automatically triggering an audible alarm and visually locating the problem point through highlighting and flashing. For example, when the equipment accessibility rate is below 80%, the water level difference rate exceeds 5%, the element arrival rate is below 90%, or the accumulated abnormal data reaches 10, the interface automatically triggers an audible alarm, and the system simultaneously pushes the alarm to the responsible person.

[0051] After generating an anomaly event marker, the routing service layer of the master node intercepts and parses the 8-bit station code of the data. By extracting the 5th and 6th bits (administrative region code) of the code, the system retrieves the pre-established "administrative region code - child node IP address / port number" mapping routing table in memory. If a match is found, the abnormal data is encapsulated into a specific communication protocol packet and sent point-to-point to the corresponding city / county child node server.

[0052] To overcome the limitations of fixed thresholds (such as a default limit of 0.2 meters) in adapting to complex hydrological environments when monitoring water level differences between main and auxiliary equipment, this solution introduces an adaptive water level difference limit calculation method based on historical fluctuations. The system extracts a dataset of historical absolute water level differences from the past 30 days when both main and auxiliary equipment were operating normally. A statistical distribution model is used to calculate the mean μ and standard deviation σ, generating a specific dynamic water level difference limit L. dynamic : L dynamic =μ+3σ; The system compares the real-time difference ΔH with dynamic limits and business baselines. When ΔH > max(0.2m, L), dynamic Only when an abnormal event is detected is an exception flag generated and added to the difference list, effectively filtering false alarms under complex flow conditions. When an indicator falls below the limit, the end-to-end monitoring interface automatically triggers an alarm to locate the source.

[0053] Step S3: Quality control rule verification and physical linkage blocking of equipment status The system establishes a quality control rule framework to verify the data, including upper and lower error limits, upper and lower doubt limits, and fluctuation limits for observed elements. Taking river station water level data as an example, the configuration rules include: water level exceeding the upper error limit is determined when the water level exceeds the levee crest elevation; water level exceeding the lower error limit is determined when the water level is below the riverbed elevation; water level fluctuation exceeding the limit in 5 minutes is determined when the water level rise exceeds 0.3m; and water level exceeding the historical highest water level is determined when the water level exceeds the upper doubt limit. City and county users can optimize thresholds for each monitoring station based on its characteristics and hydrological patterns, including station type (river station, reservoir station, weir / sluice gate station, tide station), location in the river basin, historical extreme values, flood control characteristic values ​​(warning level, guaranteed level, flood limit level), and the operational status of supporting water conservancy projects (reservoir scheduling rules, gate opening and closing patterns). For example, at a mountainous river station with significant water level fluctuations, city and county users can adjust the station's 5-minute water level fluctuation limit from the default 0.3m to 0.4m, adding the note "Mountainous river, large water level fluctuations." The optimized rule takes effect immediately and is synchronized to the provincial rule database via a vertical data stream. Each rule's modification record includes the modifier, modification time, and reason for modification, automatically saved for future reference.

[0054] The equipment status management module supports dynamic labeling of equipment as normal, faulty, or out-of-service. The dynamic labeling rules include: equipment with a short-term fault is labeled "Faulty," and equipment that cannot be repaired in the short term or has been removed is labeled "Out of Service." Water level monitoring equipment has an effective range with upper and lower limits; when the water level exceeds the range, it is automatically labeled "Abnormal." For example, if a piece of equipment malfunctions and is expected to be unrepairable within 72 hours, the user labels it "Faulty." If equipment is stranded at low water levels, the user sets the lower water level limit for that equipment to 1.5m; when the water level falls below 1.5m, the equipment's water level exceeds its effective range.

[0055] After a device is marked as "faulty" or "out of service," the following operations are automatically performed: ① The device is automatically removed from the device availability statistics and is no longer counted in the total number of valid devices; ② The device's data is no longer used in the calculation of water level difference rate; ③ The system triggers a linkage control mechanism at the underlying data gateway to intercept the data stream from the device's IP or MAC address, completely blocking the real-time data of the abnormal device from entering the application warehouse from the physical link, and the device no longer participates in any station data fusion; ④ "Faulty" devices are automatically added to the "Faulty Device List" and sorted by the number of days of failure; ⑤ "Out of service" devices are automatically added to the "Out of Service Device List" and an outage report is generated, recording the reason for the outage and the estimated recovery time; ⑥ The backup device activation process is automatically triggered to ensure uninterrupted station data.

[0056] When the water level exceeds the measurement range, the following operations will be performed automatically: ① The device will be automatically marked as "abnormal" and labeled "out of range"; ② The data from this device will no longer be used in the calculation of the water level difference rate or in the fusion of station data; ③ The device switching process will be automatically triggered, switching the backup device within the effective measurement range to the current data application device; ④ A prompt message will be generated to remind the user to check the device datum or measurement range configuration.

[0057] Step S4: Manual feedback closed loop and dynamic optimization of trigger threshold The system records the entire process from problem discovery to handling, and calculates the false positive rate and false negative rate of the quality control rules. False positive rate = number of anomalies marked as "ignore" by the user / total number of times the rule is triggered; False negative rate = number of anomalies manually corrected by the user but not recognized by the rule / total number of times the user manually corrects them.

[0058] When the false alarm rate or missed alarm rate exceeds a preset threshold, the system automatically performs dynamic optimization of the rule threshold. As a preferred optimization algorithm, taking the "amplitude exceeding limit" rule with an excessively high false alarm rate as an example: the system extracts all real hydrological feature data that triggered alarms in past periods but were marked as "ignored" by the user, forming a false alarm feature distribution set V. false_positive Calculate the 95th percentile of this set as the candidate threshold T. candidate Based on the flood control characteristic value of this monitoring station (safety limit T) maxThe system is based on the formula: T new =min(T candidate ,T max ); The final update threshold is calculated and automatically replaces the original rule threshold. For example, if a rain gauge has a false alarm rate of 90%, the system will automatically raise the threshold from 30mm to 35mm using this algorithm and synchronize it to all nodes.

[0059] When city / county users trigger a "manual correction" or "ignore" operation on the child node client, the child node program will convert it locally into a lightweight state correction command stream (e.g., encapsulated as a JSON format message containing: station ID, triggered rule ID, action type, timestamp, and operator authentication token). This command stream is asynchronously pushed into the master node's message queue through the uplink channel for the master node to consume and aggregate, completely replacing the traditional direct cross-level database write operation and ensuring system decoupling under high concurrency.

[0060] Step S5: Disaster recovery and resume transmission after network outage and bidirectional synchronization of data stream. The system packages the processed standardized dataset and optimized quality control rule set, and actively pushes them to all city and county sub-nodes through the master node client. The client records the sending status of each message and automatically writes the message to the local cache queue in case of network failure.

[0061] To address the pain point of concurrent congestion after network recovery, this invention improves the traditional sequential continuation logic by adopting a time-sensitive data queuing and dual-channel continuation mechanism: when network connectivity is restored, the system scans the local cache queue and extracts the latest data packets that are less than a preset short time window away from the current time into the "real-time priority channel" for immediate transmission; the remaining old data is kept in the "historical backup channel" and asynchronously and rate-limitedly retransmitted when the real-time priority channel is idle.

[0062] After the master node completes the threshold adaptive optimization, it serializes the updated rule parameter matrix and distributes it through the downlink channel. The daemon processes deployed on the child nodes, upon receiving the rule update message, automatically parse and hot-update / overwrite the local quality control rule configuration library file in the background, achieving millisecond-level seamless collaboration of computing power rules across all nodes in the province.

[0063] When a city or county sub-node optimizes a rule threshold on its local interface based on specific flood control characteristics (such as the unique flood discharge patterns of a reservoir), the sub-node will synchronize the customized parameter, along with the modifier, modification time, and modification reason (as a hash traceability field to prevent tampering), back to the provincial rule base of the master node. After verification, the master node will update the global mapping in real time, realizing the feedback of bottom-up edge experience to the central algorithm base.

[0064] Ultimately, the governance results are pushed to various business applications on demand through standardized API interfaces, with IP + key dual authentication implemented during the call process. Channel monitoring is deployed on master and child nodes to collect indicators such as transmission rate and backlog data in real time for alerting, achieving strong consistency distribution and collaborative sharing of hydrological data across the province.

[0065] This method has been successfully applied in Hangzhou, Lishui, Jinhua and other places in Zhejiang Province. The data flow rate has increased from 92% to over 99%, the water level difference rate at the same station has been controlled within 2%, and the time for handling abnormal data has been shortened from an average of 24 hours to less than 2 hours, significantly improving the efficiency of governance.

[0066] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0067] Although this document uses terms such as data base, channel, and exception event marker frequently, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of this invention; interpreting them as any additional limitation would contradict the spirit of this invention.

Claims

1. A hierarchical and collaborative governance method for hydrological data, characterized in that, Includes the following steps: S1: Establish a mapping relationship library of "station-equipment-element-index value" to aggregate real-time equipment data scattered in various business systems into a data base containing raw data and application data. Based on the mapping relationship library, merge the raw data of equipment level into the corresponding stations, and mark the monitoring equipment in the current data application state as the main equipment and the rest as the secondary equipment, forming a standardized dataset with main and secondary equipment identifiers in units of stations. S2: Monitor the entire data chain of the device in real time at the master node, calculate the device smoothness rate in real time with the device as the basic unit, and calculate the water level difference rate between the master device and the slave device; when the device smoothness rate or water level difference rate is detected to be worse than the preset limit, generate data with abnormal event markers as alarm data, and push the alarm data to the corresponding level of child nodes; S3: The master node uses a set of quality control rules to verify the aggregated data; at the same time, it acquires the physical objective status of the monitoring equipment in real time and dynamically marks it. When the monitoring equipment is marked as abnormal, the linkage control mechanism is triggered to block the real-time data of the abnormal equipment from entering the application warehouse. S4: The master node receives the status correction instruction stream from each level of sub-nodes based on the pushed data. The status correction instruction stream includes manual correction operations and ignore operations triggered by alarm data by users at each level. Based on the aggregated status correction instruction stream, the master node calculates the false alarm rate and false negative rate of each rule in the quality control rule set. When the false alarm rate or false negative rate exceeds the preset threshold, the trigger threshold of the corresponding rule is adjusted up or down by a certain amount. S5: Packages the standardized dataset processed by the master node and the optimized quality control rule set into a standard format message, and distributes it synchronously to sub-nodes at all levels and various business applications to achieve synchronous collaboration of data and rules across levels.

2. The method for hierarchical and collaborative management of hydrological data according to claim 1, characterized in that, The S3 also includes a bottom-up collaborative mechanism for governance rules: each level of sub-node obtains the flood control characteristic values ​​of the monitoring stations in its jurisdiction and the operation status of the supporting water conservancy projects, optimizes the thresholds of the quality control rule set station by station locally, and synchronizes the optimized rule parameters back to the provincial rule library of the master node through the vertical synchronization data stream to take effect immediately. At the same time, the master node automatically retains collaborative modification records containing the modifier, modification time and modification reason.

3. A method for hierarchical and collaborative management of hydrological data according to claim 1 or 2, characterized in that, In step S3, the linkage control mechanism specifically involves: automatically removing the monitoring device corresponding to the abnormal state from the calculation base of the device smoothness rate and the water level difference rate, blocking the real-time data of the monitoring device from entering the application warehouse, and triggering the automatic switching process of the backup device.

4. The method for hierarchical and collaborative management of hydrological data according to claim 1, characterized in that, In step S1, when multiple sets of monitoring devices at the same station report data concurrently for the same element, the backup device is automatically switched to the current data application device based on the preset effective range of the water level device and the priority of the station application device. The monitoring device in the current data application state is marked as the main device, and the other devices are marked as auxiliary devices.

5. The method for hierarchical and collaborative management of hydrological data according to claim 1, characterized in that, The formula for calculating the relevant ratio in step S2 is as follows: The equipment availability rate = (number of devices that reported data in the first hour / total number of valid devices) × 100%, where the total number of valid devices refers to the number of devices associated with the station that have not been marked as out of service or faulty by the system. The water level difference rate is calculated as follows: (Number of stations with a water level difference greater than the preset limit between the main and auxiliary equipment / Total number of stations with main and auxiliary equipment) × 100%.

6. The method for hierarchical and collaborative management of hydrological data according to claim 1, characterized in that, The formula for calculating the relevant ratio in step S4 is as follows: The false alarm rate = number of anomalies marked as "ignore" by the user / total number of times the rule is triggered; The false negative rate = the number of anomalies that were manually corrected by the user but not identified by the rules / the total number of manual corrections by the user.

7. A method for hierarchical and collaborative management of hydrological data according to claim 1 or 4, characterized in that, The station codes in the mapping database in step S1 are composed of 8-bit strings, including a 3-bit watershed code, a 1-bit station type code, a 2-bit administrative region code, and a 2-bit station serial number.

8. The method for hierarchical and collaborative management of hydrological data according to claim 1, characterized in that, The specific rules for acquiring and dynamically marking the physical objective status of the monitoring equipment in real time in step S3 include: marking equipment with short-term faults as faulty, marking equipment that cannot be repaired in a short time or has been removed as out of service, and automatically marking water level data as over-range abnormal when it exceeds the upper and lower limits of the effective range of the water level equipment.

9. The method for hierarchical and collaborative management of hydrological data according to claim 1, characterized in that, Step S1 further includes: directly aggregating manual flood reporting data, basic information of monitoring stations, and historical feature data from various business systems into the application warehouse through the second channel.

10. A method for hierarchical and collaborative management of hydrological data according to claim 1, characterized in that, Step S5 further includes: deploying channel monitoring agents on the master node and child nodes respectively, collecting in real time the connectivity status, transmission rate, backlog data volume and error rate indicators of the synchronous distribution channel, and automatically sending alarms when the channel is interrupted for more than a set time, the backlog data exceeds the threshold or the error rate exceeds the standard.

Images