Hydrological data synchronization method and device based on distributed storage and medium

Abstract

The invention discloses a hydrological data synchronization method and device based on distributed storage and a medium, and the method comprises the steps: dividing hydrological data collected by different hydrological monitoring point sensor devices in real time into a plurality of different grades according to the data emergency degree, and adding a space-time label to the hydrological data; fragmenting the hydrological data based on the space-time dimension to obtain a standard data block, and determining a regional storage node corresponding to the standard data block; according to the real-time network state, determining an optimal transmission path corresponding to different levels of hydrological data; and based on the consistency transmission rules corresponding to the multiple different levels, synchronizing the standard data blocks to the corresponding regional storage nodes through the optimal transmission path. Through level-to-level management and intelligent routing decision making of hydrological data, an optimal transmission path is selected according to characteristics of different levels of data and network states, resource waste and transmission delay caused by a traditional unified synchronization strategy are avoided, and the data synchronization efficiency is remarkably improved.

Description

Technical Field

[0001] This application relates to the field of hydrological data management, and particularly to a hydrological data synchronization method, device and medium based on distributed storage. Background Art

[0002] With the in-depth application of Internet of Things technology in the field of hydrological monitoring, millions of hydrological sensor nodes have been deployed nationwide, covering multi-dimensional monitoring indicators such as water level, flow rate, water quality, and rainfall, forming a three-dimensional monitoring network of "space-air-ground" integration. As the core basic data for water resource management, flood control and drought relief decision-making, and ecological environment monitoring, the integrity, real-time and accuracy of hydrological data directly affect the implementation effect of the national water security strategy.

[0003] In the current hydrological data management scenario, traditional data synchronization methods are mainly divided into two categories: one is the structured data synchronization method based on the database, which obtains data changes in real time by listening to the database log and supports incremental synchronization. This method is only applicable to structured monitoring data, while there are often a large amount of unstructured data in hydrological data, such as videos, radar images, etc., and it is difficult to ensure the collaborative synchronization of unstructured data by this method; the other is the unstructured data synchronization method based on the file system, which realizes differential synchronization through file hash verification and is applicable to offline or low-frequency data synchronization scenarios and can support any type of file. However, hydrological data often has strong time series characteristics, and this method lacks optimization of timestamp indexing, resulting in metadata processing becoming a bottleneck during the synchronization of a large number of small files. Summary of the Invention

[0004] To solve the above problems, this application proposes a hydrological data synchronization method based on distributed storage, including: For the hydrological data collected in real time by sensor devices at different hydrological monitoring points, divide them into multiple different levels according to the data urgency, and add spatio-temporal tags to the hydrological data; Fragment the hydrological data based on the spatio-temporal dimension to obtain standard data blocks, and determine the regional storage nodes corresponding to the standard data blocks; Determine the optimal transmission path corresponding to the hydrological data of different levels according to the real-time network status; Based on the consistency transmission rules corresponding to the multiple different levels, synchronize the standard data blocks to the corresponding regional storage nodes through the optimal transmission path.

[0005] On the other hand, this application also proposes a hydrological data synchronization device based on distributed storage, including: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to execute a method for hydrological data synchronization based on distributed storage as described in the above example.

[0006] On the other hand, the present application also proposes a non-volatile computer storage medium storing computer-executable instructions, and the computer-executable instructions are configured as: a method for hydrological data synchronization based on distributed storage as described in the above example.

[0007] The method for hydrological data synchronization based on distributed storage proposed by the present application can bring the following beneficial effects: By performing hierarchical management and intelligent routing decision on hydrological data, and selecting the optimal transmission path according to the characteristics of different-level data and network status, it avoids the resource waste and transmission delay caused by the traditional unified synchronization strategy, significantly improves the efficiency of data synchronization, especially for real-time warning data, which can be transmitted quickly and accurately, providing timely support for flood control decision-making.

[0008] The hierarchical consistency processing strategy adopts appropriate consistency protocols according to the requirements of different-level data, ensuring the accuracy and reliability of each level of data during the synchronization process. The strong consistency protocol guarantees the consistency of real-time warning data and avoids incorrect decisions caused by inconsistent data; the eventual consistency and weak consistency protocols reduce the consumption of system resources while meeting the corresponding data requirements, ensuring the long-term consistency of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation to the present application. In the drawings: Figure 1 is a schematic flowchart of a method for hydrological data synchronization based on distributed storage in an embodiment of the present application; Figure 2 is a schematic diagram of a device for hydrological data synchronization based on distributed storage in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0010] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0011] The following will describe in detail the technical solutions provided by each embodiment of the present application in conjunction with the accompanying drawings.

[0012] As Figure 1 shown, an embodiment of the present application provides a hydrological data synchronization method based on distributed storage, including: S101. For the hydrological data collected in real time by sensor devices at different hydrological monitoring points, divide it into multiple different levels according to the data urgency level, and add spatio-temporal tags to the hydrological data.

[0013] Specifically, deploy multiple edge storage nodes at hydrological monitoring points, collect hydrological data in real time through sensor devices, and store it in the edge storage nodes. Divide the hydrological monitoring area into multiple basin areas according to the longitude and latitude coordinates of the hydrological monitoring points, and build a distributed storage cluster based on the edge storage nodes in the basin areas.

[0014] Further, collect hydrological data in real time through sensor devices, cut the hydrological data into several data blocks and several check blocks, and distribute the several data blocks to different edge storage nodes to build multiple replicated data corresponding to the hydrological data.

[0015] Utilize sensors distributed in different geographical locations to collect hydrological data in real time. According to the importance and real-time requirements of the data, divide the data into first-level real-time warning data, second-level regular monitoring data, and third-level historical archived data. Among them, the first-level data is real-time warning data updated at the second level during the flood period, and is used in scenarios with extremely high real-time requirements such as flood control; the second-level data is regular monitoring data updated at the minute level and is suitable for daily hydrological monitoring; the third-level data is historical archived data updated at the hour level and is mainly used for historical data analysis and archiving. At the same time, add spatio-temporal tags containing timestamps, geographical locations, and monitoring device identifiers to each data item for subsequent data management and analysis.

[0016] S102. Shard the hydrological data based on the spatio-temporal dimension to obtain standard data blocks, and determine the regional storage nodes corresponding to the standard data blocks.

[0017] Specifically, according to the spatio-temporal tags, determine the collection timestamp and geographical space coordinates of the hydrological data, generate a spatio-temporal hash sharding key based on the collection timestamp and geographical space coordinates, and shard the hydrological data into standard data blocks through the spatio-temporal hash sharding key. According to the spatio-temporal hash sharding key, determine the regional storage nodes corresponding to the standard data blocks.

[0018] Further, according to the spatial encoding in the spatio-temporal hash sharding key, determine the sharding storage location corresponding to the standard data block, determine the regional storage node where the sharding storage location is located, and build a local location index of the standard data block based on the sharding storage location and the regional storage node.

[0019] It should be noted that the spatio-temporal hash sharding key is a composite hash key that combines geospatial coordinates and the time dimension, and is used to determine the physical location of data storage in a distributed storage system. Its core purpose is to allocate hydrological data with spatio-temporal correlation (such as monitoring data within a specific time period in a certain basin) to the same or adjacent storage nodes, thereby improving the locality of data processing and query efficiency.

[0020] In the embodiments of the present application, the data with spatio-temporal tags added is sharded according to the time and space dimensions to form standard data blocks. Second-level time granularity sharding is adopted during the flood season, and minute-level time granularity sharding is adopted during the normal water season, and sharding is combined with spatial regions such as basins or provinces. The regional node establishes a local index for each data shard, recording its corresponding monitoring point ID, time range, and storage location information; the central hub node aggregates the local indexes of each regional node to construct a global index. Through data sharding and index construction, hydrological data at all levels can be efficiently managed, laying a solid foundation for subsequent intelligent routing decisions and data synchronization processes.

[0021] S103. Determine the optimal transmission path corresponding to the hydrological data of different levels according to the real-time network state.

[0022] Specifically, network state information is obtained in real time. The network state information includes link delay, bandwidth, and node load. According to the network state information, the optimal transmission path corresponding to the hydrological data of different levels is determined.

[0023] In the embodiments of the present application, network state information is obtained in real time, including link delay, bandwidth, and node load conditions. According to data classification and network state, transmission paths are selected for different levels of data: level-1 data is preferentially transmitted in real time through a low-latency link (such as a 5G private network) to ensure the timeliness of real-time early warning; level-2 and level-3 data are transmitted asynchronously or in batches through a high-latency link (such as a satellite link) or during network idle periods, so as to rationally utilize network resources. In addition, the intelligent routing decision also includes optimizing the transmission path according to historical transmission quality data, and preferentially selecting a link with a high transmission success rate and stable delay to further improve the efficiency and reliability of data transmission.

[0024] Execute different consistency strategies for different levels of data. For first-level data, adopt a strong consistency protocol, specifically an improved Raft protocol, to synchronize data replicas among at least three nodes within the region, and perform data synchronization through a 10 Gigabit network between data centers within the region to ensure that the synchronization latency is less than 500 ms, meeting the strict requirements for data accuracy and timeliness in real-time warning; for second-level data, adopt a final consistency protocol, allowing a short delay, and when version conflicts occur, take the data with the latest timestamp as the standard, while ensuring the final consistency of the data, reducing resource consumption during the synchronization process; for third-level data, adopt a weak consistency protocol, only perform local storage, and conduct batch reconciliation processing daily to balance data consistency and the utilization of storage resources.

[0025] S104. Based on the consistency transmission rules corresponding to the multiple different levels, synchronize the standard data block to the corresponding regional storage node through the optimal transmission path.

[0026] Specifically, execute different consistency strategies for different levels of data: for first-level data, adopt a strong consistency protocol to synchronize data replicas among at least three nodes within the region to ensure that the synchronization latency is less than 500 ms; for second-level data, adopt a final consistency protocol, allowing a short delay, and when version conflicts occur, take the data with the latest timestamp as the standard; for third-level data, adopt a weak consistency protocol, only perform local storage, and conduct batch reconciliation processing daily. Through hierarchical consistency processing, hydrological data at different levels can not only meet their respective timeliness and accuracy requirements during the synchronization process, but also achieve an efficient balance in the utilization of storage and transmission resources, providing a stable and reliable data state basis for subsequent closed-loop optimization and feedback. Update the status information of the regional storage node. The status information includes node load, storage capacity, and link quality. According to the updated status information, optimize the transmission path for the next data synchronization. After the next data synchronization is completed, update the status information of each node, including node load, storage capacity, and link quality; according to the updated status information, optimize the routing strategy, transmission method, and consistency processing method for the next data synchronization to form a dynamic optimization closed loop.

[0027] It should be noted that the consistency strategy refers to the rules and methods adopted in a distributed system to ensure the consistency of data on multiple nodes during the processes of update, storage, and transmission. For the first-level real-time warning data used in critical scenarios such as flood warnings, a strong consistency protocol is adopted to quickly synchronize data replicas among multiple nodes, ensuring the consistency of data on each node and avoiding false warnings caused by inconsistent data, providing accurate and reliable data support for flood control decision-making. The second-level regular monitoring data adopts the eventual consistency protocol, allowing for a short delay in data. On the premise of meeting the requirements of daily monitoring operations, it reduces the resource consumption caused by frequent synchronization and improves the overall operation efficiency of the system. The third-level historical archived data adopts the weak consistency protocol, only storing locally and reconciling periodically, concentrating resources on data storage and verification, avoiding unnecessary real-time synchronization overhead, and reasonably allocating system resources. The consistency strategies for different levels of data ensure the reliable processing of data in various business scenarios, reduce the risks of data loss, errors, or inconsistencies, enhance the stability and reliability of the entire hydrological data management system, and enable it to better cope with complex monitoring environments and business requirements.

[0028] In the embodiments of this application, different consistency strategies are executed for different levels of data. For the first-level data, a strong consistency protocol is adopted, specifically the improved Raft protocol, to synchronize data replicas among at least three nodes within the region. Data synchronization is carried out through the 10 Gigabit network between data centers within the region, ensuring that the synchronization delay is less than 500 ms, meeting the strict requirements of real-time warning for data accuracy and timeliness. For the second-level data, the eventual consistency protocol is adopted, allowing for a short delay. When a version conflict occurs, the data with the latest timestamp is used as the standard, reducing resource consumption during the synchronization process while ensuring the eventual consistency of the data. For the third-level data, the weak consistency protocol is adopted, only storing locally and performing batch reconciliation processing daily to balance data consistency and the utilization of storage resources.

[0029] By hierarchically managing hydrological data and making intelligent routing decisions, selecting the optimal transmission path according to the characteristics of different levels of data and network status, it avoids the resource waste and transmission delay caused by the traditional unified synchronization strategy, significantly improving the efficiency of data synchronization. Especially for real-time warning data, it can be transmitted quickly and accurately, providing timely support for flood control decision-making.

[0030] The hierarchical consistency processing strategy adopts appropriate consistency protocols according to the requirements of different levels of data, ensuring the accuracy and reliability of each level of data during the synchronization process. The strong consistency protocol guarantees the consistency of real-time warning data and avoids incorrect decisions caused by inconsistent data. The eventual consistency and weak consistency protocols reduce system resource consumption while meeting the corresponding data requirements, ensuring the long-term consistency of the data.

[0031] Such asFigure 2 As shown in Figure 2 , an embodiment of the present application further provides a hydrological data synchronization device based on distributed storage, including: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a hydrological data synchronization method based on distributed storage as described in any one of the above embodiments.

[0032] The various embodiments in the present application are all described in a progressive manner. The same or similar parts among the various embodiments can be referred to each other, and the key points of each embodiment are the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments.

[0033] The devices and media provided by the embodiments of the present application correspond one by one to the methods. Therefore, the devices and media also have beneficial technical effects similar to those of the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be elaborated here.

[0034] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0035] The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0036] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one or more of the processes Figure 1 one or more processes and / or blocks Figure 1 specified in the block or blocks.

[0037] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are performed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the processes Figure 1 one or more processes and / or blocks Figure 1 specified in the block or blocks.

[0038] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0039] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0040] Computer-readable media includes both permanent and non-permanent, removable and non-removable media implemented by any method or technology for storing information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0041] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or apparatus comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

[0042] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A hydrological data synchronization method based on distributed storage, characterized in that, Including: For the hydrological data collected in real time by sensor devices at different hydrological monitoring points, divide it into multiple different levels according to the urgency of the data, and add spatio-temporal tags to the hydrological data; Slice the hydrological data based on the spatio-temporal dimension to obtain standard data blocks, and determine the regional storage nodes corresponding to the standard data blocks; Determine the optimal transmission path corresponding to the hydrological data of different levels according to the real-time network status; Based on the consistency transmission rules corresponding to the multiple different levels, synchronize the standard data blocks to the corresponding regional storage nodes through the optimal transmission path.

2. The method for synchronizing hydrological data based on distributed storage according to claim 1, wherein Before dividing the hydrological data collected in real time by sensor devices at different hydrological monitoring points into multiple different levels according to the urgency of the data, the method further includes: Deploy multiple edge storage nodes at hydrological monitoring points; Collect hydrological data in real time through sensor devices and store it in the edge storage nodes; According to the longitude and latitude coordinates of the hydrological monitoring points, divide the hydrological monitoring area into multiple watershed areas, and construct a distributed storage cluster based on the edge storage nodes in the watershed areas.

3. A method for hydrological data synchronization based on distributed storage according to claim 2, characterized in that, The collecting hydrological data in real time through sensor devices and storing it in the edge storage nodes specifically includes: Collect hydrological data in real time through sensor devices, and cut the hydrological data into several data blocks and several check blocks; Distribute and store the several data blocks in different edge storage nodes to construct multiple replicated data corresponding to the hydrological data.

4. A method for synchronizing hydrological data based on distributed storage according to claim 2, characterized in that, The method further includes: When data synchronization is interrupted, temporarily store the standard data blocks stored in the edge storage nodes locally and record a difference log; When data synchronization is restored, determine the difference data between the regional storage nodes and the edge data nodes according to the difference log, and synchronize the difference data to the regional storage nodes.

5. A method for hydrological data synchronization based on distributed storage according to claim 1, characterized in that, The slicing the hydrological data based on the spatio-temporal dimension to obtain standard data blocks, and determining the regional storage nodes corresponding to the standard data blocks specifically includes: According to the spatio-temporal tags, determine the collection timestamp and geographical space coordinates of the hydrological data; Generate a spatio-temporal hash slicing key based on the collection timestamp and the geographical space coordinates, and slice the hydrological data into standard data blocks through the spatio-temporal hash slicing key; According to the spatio-temporal hash slicing key, determine the regional storage nodes corresponding to the standard data blocks.

6. A method for hydrological data synchronization based on distributed storage according to claim 5, characterized in that The determining the regional storage nodes corresponding to the standard data blocks according to the spatio-temporal hash slicing key, According to the space encoding in the spatio-temporal hash slicing key, determine the sliced storage location corresponding to the standard data block; Determine the regional storage node where the sliced storage location is located, and construct a local location index of the standard data block based on the sliced storage location and the regional storage node.

7. A method for hydrological data synchronization based on distributed storage according to claim 1, characterized in that, The multiple different levels include level 1 real-time warning, level 2 regular monitoring, and level 3 historical archiving; The determining the optimal transmission path corresponding to the hydrological data of different levels according to the real-time network status specifically includes: Obtain network status information in real time; the network status information includes link delay, bandwidth, and node load; Determine an optimal transmission path corresponding to the hydrological data of different levels according to the network status information.

8. A method for hydrological data synchronization based on distributed storage according to claim 1, characterized in that, After synchronizing the standard data block to the corresponding regional storage node through the optimal transmission path, the method further includes: Updating the status information of the regional storage node; the status information includes node load, storage capacity, and link quality; Optimize the transmission path for the next data synchronization according to the updated status information.

9. A hydrological data synchronization device based on distributed storage, characterized in that, Includes: At least one processor; And, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a hydrological data synchronization method according to any one of claims 1 to 8.

10. A non-volatile computer storage medium stores computer-executable instructions, characterized in that, The computer-executable instructions are set to a hydrological data synchronization method according to any one of claims 1 to 8.

Images