A land and resource planning data storage method, system, device and medium

Abstract

The application relates to the technical field of land and resource data processing, in particular to a land and resource planning data storage method, system, device and medium, which comprises the following steps: carrying out fragmentation processing on land and resource data to obtain data fragmentation information; carrying out real-time analysis on the data fragmentation information to obtain hot data and cold data, carrying out synchronous replication processing on the hot data, and carrying out asynchronous replication processing on the cold data; obtaining memory state information, dividing the memory state information to obtain hierarchical information, adjusting the hierarchical information according to the hot data, and storing the hot data into a plurality of hierarchical caches; obtaining data access traffic information in real time, dynamically adjusting the data fragmentation information according to the data access traffic information to obtain distribution update information for updating the hot data and the cold data, enabling the land and resource to be stably stored, improving the consistency of the data, and thus improving the reliability of the system and the user experience.

Description

Technical Field

[0001] This application relates to the field of land and resources data processing technology, and in particular to a data storage method, system, device and medium for land and resources planning. Background Technology

[0002] Land and resources information plays a crucial role in modern society, encompassing land use, resource exploration, environmental protection, and urban and rural planning. With rapid economic development and accelerated urbanization, the complexity and volume of land and resources management are constantly increasing, raising the requirements for information storage and management. This information not only needs efficient storage but also flexible data processing capabilities. In traditional data storage methods, data is typically stored centrally in a single database or storage system. While this centralized storage method simplifies data management to some extent, as data volume grows, single storage systems face performance bottlenecks and scalability issues. To address these problems, distributed storage systems are adopted for storing land and resources information.

[0003] Distributed storage systems, by distributing data across multiple nodes, not only increase the capacity and processing power of the storage system but also improve data availability and reliability. In such systems, data is divided into multiple fragments, each stored on a different node. This allows the system to expand its storage capacity and computing power by adding nodes. However, while achieving these advantages, distributed storage systems also present challenges. One of the most significant issues is data consistency. Data consistency problems arise when multiple nodes operate on the same data simultaneously, raising the question of how to ensure that the data on all nodes eventually remains consistent. In distributed systems, because data is stored across multiple nodes, network latency and failures between nodes can lead to discrepancies between data replicas, thus causing data consistency issues.

[0004] When multiple nodes modify data simultaneously, data consistency cannot be guaranteed, leading to inconsistent data replicas, incorrect query results, conflicts and data loss when the system handles concurrent operations, and discrepancies between the data obtained by users and reality. This affects the accuracy of decision-making, and consequently the reliability of the system and the user experience. These issues need to be addressed. Summary of the Invention

[0005] To ensure stable storage of land and resources data, improve data consistency, and thus enhance system reliability and user experience, this application provides a data storage method, system, device, and medium for land and resources planning, employing the following technical solution:

[0006] Firstly, this application provides a data storage method for land and resources planning, comprising:

[0007] Land and resources data are segmented to obtain data segmentation information;

[0008] Real-time analysis of data sharding information is performed to identify hot and cold data. Hot data is synchronously replicated, while cold data is asynchronously replicated.

[0009] Obtain memory status information, divide the memory status information to obtain hierarchical information, adjust the hierarchical information according to hot data, and store hot data in several hierarchical caches;

[0010] Real-time data access traffic information is acquired, and data sharding information is dynamically adjusted based on the data access traffic information to obtain distribution update information for updating hot data and cold data.

[0011] Preferably, the specific steps for synchronously copying hot data are as follows:

[0012] Obtain replication factor information and hotspot synchronization frequency information, replicate hotspot data several times based on replication factor information, and adjust the replication rate of hotspot data based on hotspot synchronization frequency information.

[0013] Preferably, the specific steps for asynchronously copying cold data are as follows:

[0014] Obtain replication factor information and cold synchronization frequency information, replicate cold data a number of times based on replication factor information, and adjust the replication rate of cold data based on cold synchronization frequency information.

[0015] Preferred options also include:

[0016] Real-time acquisition of operational indicator information, analysis of operational indicator information, and obtaining operational status information;

[0017] If the running status information indicates an abnormal mode, a backup copy is built based on the hierarchical cache, and hot data and cold data are transferred and stored to the backup copy.

[0018] Preferred options also include:

[0019] The number of backup copies is dynamically adjusted based on data access traffic information and operational metrics. Specifically, if the operational status information indicates an abnormal mode, the number of backup copies of the current hot data is increased, and if the operational status information indicates a normal mode, the number of backup copies of the current cold data is decreased.

[0020] Preferred options also include:

[0021] Obtain vector clock information and change information of several nodes. Mark the change information with timestamps according to the vector clock information to obtain change record information used to record the order of change.

[0022] Preferred options also include:

[0023] Obtain node monitoring status information. If the node monitoring status information indicates a node failure, obtain the hot and cold data of the backup copy and restore the hot and cold data of the backup copy to the hierarchical cache corresponding to the node failure.

[0024] Secondly, this application provides a data storage system for land and resources planning, comprising:

[0025] The information acquisition module is used to acquire specified land and resources storage data, perform data segmentation processing on the stored data, and obtain data segmentation information.

[0026] The sharding analysis module is used to perform real-time analysis of data sharding information to obtain hot data and cold data. Hot data is synchronously copied, while cold data is asynchronously copied.

[0027] The partitioning module is used to obtain memory status information, partition the memory status information to obtain hierarchical information, adjust the hierarchical information according to hot data, and store hot data into several hierarchical caches.

[0028] The dynamic adjustment module is used to acquire data access traffic information in real time, and dynamically adjust the data sharding information based on the data access traffic information to obtain distribution update information for updating hot data and cold data.

[0029] Thirdly, this application provides a data storage device for land and resources planning, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the data storage method for land and resources planning as described above.

[0030] Fourthly, this application provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the data storage method for land and resources planning as described above when it is run.

[0031] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following:

[0032] This application processes the stored data of designated land and resources, analyzes the data fragmentation information to obtain hot and cold data, employs strong consistency synchronous replication for hot data, and eventual consistency asynchronous replication for cold data. After processing, it acquires memory state information and divides the data into several hierarchical levels. The hierarchical information is adjusted using hot data, thereby storing hot data in several hierarchical caches. Real-time data access traffic information is then acquired, and the data fragmentation information is dynamically adjusted based on this information to obtain distributed update information for updating hot and cold data. By adjusting the data fragmentation distribution strategy in real time using this distributed update information, the land and resources data can be stored stably, improving data consistency and thus enhancing system reliability and user experience. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating a data storage method for land and resources planning as described in an embodiment of this application.

[0034] Figure 2 This is a schematic diagram of a data storage system for land and resources planning as described in an embodiment of this application.

[0035] Figure 3 This is a schematic diagram of the data storage system described in the embodiments of this application.

[0036] Figure 4 This is a schematic diagram of the structure of a data storage device for land and resources planning as described in an embodiment of this application.

[0037] Explanation of reference numerals in the attached figures:

[0038] 1. Information Acquisition Module; 2. Sharding Analysis Module; 3. Partitioning Module; 4. Dynamic Adjustment Module; 5. Status Analysis Module; 6. Replica Adjustment Module; 7. Clock Signal Acquisition Module; 8. Node Monitoring Module; 9. Global Transaction Module. Detailed Implementation

[0039] The following combination Figures 1-4 The present application will be described in further detail below. The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting.

[0040] Reference Figure 1 The data storage method for land and resources planning involved in this application specifically includes:

[0041] Step S1: Perform data segmentation processing on the land and resources data to obtain data segmentation information;

[0042] Step S2: Perform real-time analysis on the data sharding information to obtain hot data and cold data, perform synchronous replication on the hot data, and perform asynchronous replication on the cold data;

[0043] Step S3: Obtain memory status information, divide the memory status information to obtain hierarchical information, adjust the hierarchical information according to hot data, and store hot data in several hierarchical caches;

[0044] Step S4: Obtain data access traffic information in real time, and dynamically adjust the data sharding information based on the data access traffic information to obtain distribution update information for updating hot data and cold data.

[0045] Specifically, this application designs a flexible data consistency scheme that can dynamically adjust the level of data consistency according to different business scenarios and needs. It acquires scenario characteristic information, i.e., preset conditions, and selects corresponding land and resources storage data based on these characteristics. The stored data is then preprocessed, and a representative sample is extracted from the massive dataset for preliminary analysis and processing. By processing the stored land and resources data, the data sharding information is analyzed to identify hot and cold data. Hot data is processed using strong consistency synchronous replication, while cold data is processed using eventual consistency asynchronous replication. After processing, memory state information is acquired and partitioned to obtain hierarchical information at several levels. The hierarchical information is adjusted using hot data, thereby storing hot data in several hierarchical caches. Real-time data access traffic information is then acquired, and the data sharding information is dynamically adjusted based on this information to obtain distributed update information for updating hot and cold data. This distributed update information is used to adjust the data sharding distribution strategy in real time, ensuring stable storage of land and resources data, improving data consistency, and thus enhancing system reliability and user experience.

[0046] In large-scale distributed systems, data consistency is a critical issue, especially during data sharding and storage. In this application's embodiments, "hot data" refers to data that is frequently accessed and modified. In distributed systems, hot data is the focus of user requests or data subject to high-frequency operations. "Cold data" refers to data with low access frequency and few modification operations. This data includes historical data, archived data, or infrequently accessed secondary data.

[0047] The criteria for analyzing and judging hot and cold data are whether the access frequency and importance of the data meet the classification standards.

[0048] Due to frequent access and modification operations, hot data often encounters high concurrency requests. Since frequent changes in data mean that any delay or inconsistency can have a serious impact on user experience or business logic, the consistency requirements for hot data are very high.

[0049] This application embodiment extracts metadata from land and resources data from the storage system. The metadata includes data fragmentation information, such as fragment location and size. The data fragments are preprocessed to extract useful information and format them. Real-time analytics tools, such as Apache Kafka, Apache Flink, or Spark Streaming, are used to identify hot data, which refers to data with high access frequency or importance. Specific access rate thresholds or importance thresholds are set for identification.

[0050] The synchronous replication process used in this application is a strong consistency strategy. For hot data, a strong consistency strategy is needed to ensure that the data on all nodes remains consistent. For example, a distributed consensus algorithm based on the Paxos protocol can be used to handle high-concurrency write requests. The Paxos protocol improves data consistency through majority voting, enabling data consistency to be maintained even in the event of node failure.

[0051] Cold data is data that is accessed infrequently, and it is identified using the same analytical methods as hot data.

[0052] Cold data is accessed far less frequently than hot data, meaning it is not read or modified often. Due to the low access frequency, there are relatively few concurrent operations on cold data. Therefore, the consistency requirements for cold data are more lenient; eventual consistency is usually sufficient to meet business needs, allowing more device resources to be allocated and used for processing hot data.

[0053] For cold data, an eventual consistency strategy is used to handle data synchronization. Eventual consistency allows for inconsistencies in data for a short period, but eventually a consistent state is reached. For example, a distributed log synchronization mechanism based on the Raft protocol can be used to achieve data synchronization between nodes by recording logs and playing them sequentially. Leveraging the inherent characteristics of the protocols, Paxos is used to handle high-concurrency write scenarios, while Raft is suitable for low-frequency update data synchronization, allowing for more rational resource allocation and thus improving the system's user experience.

[0054] As one implementation method, the specific steps for synchronously copying hot data are as follows:

[0055] Obtain replication factor information and hotspot synchronization frequency information, replicate hotspot data several times based on replication factor information, and adjust the replication rate of hotspot data based on hotspot synchronization frequency information.

[0056] Specifically, the replication factor information refers to the number of copies of data that are replicated in the system. For example, if the replication factor is 3, then each piece of data will be stored in 3 copies on different nodes.

[0057] The system retrieves the current replication factor settings from configuration management or metadata storage. The replication factor is adjusted based on the importance and access frequency of the data. Hot data requires a higher replication factor to ensure high availability and fast access.

[0058] The hotspot synchronization frequency information refers to the time interval for updating hotspot data replicas. For example, if the synchronization frequency is once per minute, the system will analyze the data shard information every minute, thus achieving the effect of updating all hotspot data replicas every minute.

[0059] The system obtains the synchronization frequency settings from configuration management or business rules. The synchronization frequency is adjusted according to data update requirements to balance consistency and performance needs.

[0060] Based on the replication factor, frequently accessed data is replicated from the primary storage node to multiple backup nodes. The replication process typically includes data transfer and write operations to ensure consistency across all replicas. For example, if the replication factor for frequently accessed data is set to 3, the system will replicate the data from the primary node to two other nodes.

[0061] The replication rate is adjusted based on the hotspot synchronization frequency. For example, if the synchronization frequency is once per minute, the system may need to complete data replication and synchronization within one minute. The system can use flow control and load balancing techniques to optimize the replication rate, ensuring stable data synchronization even under high load.

[0062] In one embodiment, hotspot data in a GIS system includes the latest geological survey results and environmental monitoring data. To ensure users receive the latest geological data during real-time queries, the system needs to efficiently synchronize and replicate this data.

[0063] Set the replication factor of hot data to 5 so that each piece of hot data has a copy on 5 different nodes;

[0064] The synchronization frequency is set to once every 10 minutes because geological data does not change frequently but needs to maintain a high degree of consistency.

[0065] After receiving new geological data updates, the master node distributes the data to the five nodes.

[0066] The system checks and synchronizes data replicas on all nodes every 10 minutes to improve consistency;

[0067] If the system detects that some nodes are under high load, the replication process will automatically adjust the rate to avoid overload.

[0068] In another embodiment, within an urban planning system, hotspot data includes the latest building permit applications and urban development updates. This data needs to be synchronized rapidly to support real-time planning decisions.

[0069] Set the replication factor of hot data to 3 so that the data is backed up on 3 nodes;

[0070] The synchronization frequency is set to once every 5 minutes to keep the data up-to-date in a rapidly changing urban development environment;

[0071] After receiving a new license request, the primary storage node replicates the data to the three backup nodes;

[0072] The system checks data consistency every 5 minutes and performs synchronization operations when necessary.

[0073] If insufficient network bandwidth is detected, the system will adjust the data replication rate to optimize performance and ensure the stability of data synchronization.

[0074] The steps described above illustrate how to efficiently manage and synchronize hot data by configuring replication factors and synchronization frequencies. This method improves the high availability and consistency of hot data while dynamically adjusting data replication and synchronization strategies based on actual needs.

[0075] As one implementation method, the specific steps for asynchronously copying cold data are as follows:

[0076] Obtain replication factor information and cold synchronization frequency information, replicate cold data a number of times based on replication factor information, and adjust the replication rate of cold data based on cold synchronization frequency information.

[0077] Specifically, the replication factor information in this application embodiment is the same as the information obtained during hot data processing, but the replication factor information of hot data and cold data is not limited to the same value.

[0078] The cold synchronization frequency information refers to the frequency of synchronization updates for cold data, which determines the synchronization time interval between replicas. For example, a cold synchronization frequency of once a day means synchronization occurs once every 24 hours.

[0079] Based on a preset replication factor, cold data is replicated to multiple storage locations, involving the creation of multiple replicas to be stored on different storage devices or nodes. For example, if the replication factor is 3, the cold data is replicated to three different storage devices.

[0080] The system adjusts the data replication rate and frequency based on preset synchronization frequency information. For example, if the synchronization frequency is once a day, the system will be set to perform data replication operations daily, and execute them during off-peak hours to reduce the impact on system performance.

[0081] In one embodiment, the replication factor is set to 3, meaning that each copy of cold data will be replicated to 3 different storage locations. The cold synchronization frequency is set to once a day, and the copies of cold data are synchronized and updated every 24 hours.

[0082] Assuming the cold data is "geological survey data," this data needs to be copied three times. The system copies the original data to three different hard drives or storage nodes to ensure sufficient high availability and redundancy.

[0083] Based on the cold synchronization frequency, the system automatically performs cold data synchronization operations at a specific time each day, such as 2:00 AM. To reduce the impact on system performance during peak hours, the replication process is performed during off-peak hours, and appropriate bandwidth control strategies are used to optimize the replication process.

[0084] The system continuously monitors the replication status and synchronization of cold data. If any delay or failure in the synchronization of any replica is detected, the system will automatically trigger an alarm and take measures such as retrying replication or readjusting the replication rate. Through these steps, the system enables reliable storage and efficient access to cold data, while reducing performance issues caused by frequent synchronization.

[0085] As one implementation method, memory state information is obtained, the memory state information is divided to obtain hierarchical information, the hierarchical information is adjusted according to hot data, and the hot data is stored in several hierarchical caches.

[0086] Specifically, in large-scale distributed systems, the efficiency of data access and the system's response speed are often limited by data storage latency and access speed. The embodiments of this application provide an efficient caching architecture that can significantly improve system performance.

[0087] The system acquires memory state information related to the storage device, divides this information into hierarchical levels. These levels include local cache, distributed cache, and persistent storage. Based on this information, the corresponding external hardware is then adjusted.

[0088] Local caching refers to a cache maintained on each compute node or server to store frequently accessed copies of data on the node. It resides in memory and interacts directly with application processes or the operating system. Because data is stored directly in local memory, local caching results in extremely low data access latency, reduces access to remote nodes, and lowers network bandwidth consumption.

[0089] A distributed cache is a caching system spanning multiple computing nodes, consisting of multiple cache nodes. It's used to share cached data throughout the system, improving the availability and consistency of data access. Specifically, it distributes cached data across multiple cache nodes to balance the access load. A distributed cache system has a redundancy mechanism; when one node fails, other nodes can continue to provide service. Cache nodes can be added as needed to handle larger data volumes and higher access loads.

[0090] Persistent storage refers to storing data on stable storage media, such as hard drives, SSDs, or database systems. It is used for long-term data storage and retrieval. Data will not be lost due to node failures or restarts, thus improving long-term data reliability.

[0091] As one implementation method, data access traffic information is acquired in real time, and data sharding information is dynamically adjusted based on the data access traffic information to obtain distribution update information for updating hot data and cold data.

[0092] Specifically, embodiments of this application monitor data access traffic in real time and adjust the data sharding distribution strategy based on prediction results. Specifically, this is achieved through dynamic adjustments using load balancing algorithms and cache management strategies, ensuring that hot data is distributed appropriately, optimizing cache utilization and data consistency.

[0093] This application employs real-time monitoring tools to track the access traffic of each data segment, including read and write operations. The collected data traffic information is processed to identify hot data (high access frequency) and cold data (low access frequency), with the analysis results including trends and patterns in data access volume. Based on the traffic analysis results, the data sharding strategy is dynamically adjusted. For hot data, data replicas are increased and distributed across more nodes to reduce load; for cold data, the number of replicas is reduced to optimize storage resources. A sharding redistribution mechanism is implemented in the distributed system, applying the new sharding configuration to the system. This ensures data consistency and availability during the data sharding adjustment process.

[0094] Monitor system performance after sharding adjustments and evaluate the effectiveness of handling hot and cold data. Continuously optimize and adjust strategies based on feedback data to improve overall system performance and data consistency.

[0095] As one implementation method, it also includes:

[0096] Real-time acquisition of operational indicator information, analysis of operational indicator information, and obtaining operational status information;

[0097] If the running status information indicates an abnormal mode, a backup copy is built based on the hierarchical cache, and hot data and cold data are transferred and stored to the backup copy.

[0098] The operational metrics include CPU utilization, memory usage, disk information, and network traffic.

[0099] Specifically, this application embodiment improves the system's fault tolerance through mechanisms such as data redundancy and failover to cope with various abnormal situations and further enhance the consistency of system data. The operational metrics information includes real-time acquisition of device CPU utilization, memory usage, disk I / O, and network traffic information.

[0100] By employing time series analysis and clustering algorithms, real-time operational metrics such as CPU utilization, memory usage, disk I / O, and network traffic of the acquired devices are analyzed to collect operational status information. Based on this information, the current status is determined to be either normal or abnormal. If an abnormal mode is identified, data migration and replica creation operations are triggered, working in conjunction with the aforementioned data distribution strategy.

[0101] When the system detects an abnormal pattern, it triggers a data migration operation. Specifically, hot and cold data are moved to backup copies to reduce the load on the primary system and protect the data.

[0102] The system dynamically adjusts the number of replicas based on data importance, access frequency, and current load. Under normal circumstances, the number of replicas for cold data is reduced to save resources; during fault warnings, the number of replicas for critical data is increased to ensure data security and availability.

[0103] As one implementation method, the number of backup copies is dynamically adjusted based on data access traffic information and operation indicator information. Specifically, when the operation status information indicates an abnormal mode, the number of backup copies of the current hot data is increased, and when the operation status information indicates a normal mode, the number of backup copies of the current cold data is decreased.

[0104] Specifically, to optimize system performance and data availability, this application implements a strategy for dynamically adjusting the number of backup replicas. This strategy adjusts the number of replicas for different data shards based on data access traffic information, operational metrics, and system anomalies. This strategy improves the overall system performance, load balancing capabilities, concurrency processing capabilities, fault tolerance, and data consistency.

[0105] Specifically, the system dynamically adjusts the number of replicas for different data shards. This adjustment strategy takes into account the importance of the data, the frequency of access, and the current system load.

[0106] Frequently accessed data has an increased number of replicas, improving availability under high concurrency. Conversely, fewer replicas of cold data reduce storage and synchronization overhead, thus optimizing system performance.

[0107] This application's embodiments adjust the number of replicas based on data importance and access frequency. For example, transaction records and user data are marked as critical data, so the number of replicas is increased first to improve data availability and system fault tolerance.

[0108] Furthermore, data with high access frequency will have more replicas to meet high concurrency requirements, while data with low access frequency will have fewer replicas to save storage and network resources.

[0109] Furthermore, during normal operation, the system adjusts the number of replicas based on the load to balance storage resources and data synchronization overhead. Under high load, the number of replicas for important data is increased, while under low load, the number of replicas for cold data is reduced.

[0110] This application's embodiments can improve system performance by reducing the number of replicas of cold data, thereby reducing data synchronization and storage overhead and improving overall system performance. Dynamically adjusting the number of replicas can effectively distribute the load, optimize the use of system resources, and achieve load balancing. Increasing the number of replicas of hot data improves the system's ability to handle high-concurrency access. Furthermore, increasing the number of replicas of critical data enhances the system's resilience to failures. It also allows for reasonable adjustment of the number of replicas, ensuring data consistency and reliability across different replicas. By dynamically adjusting the number of replicas, data can always be accessed quickly and stored reliably under different loads and operating conditions.

[0111] The system contains numerous data shards distributed across different physical nodes, with each shard having multiple replicas. When data in a shard changes, the changes need to be quickly and accurately synchronized to other replicas, ensuring that all replicas of the same shard eventually reach a consistent state. However, in complex network environments, data transmission latency varies significantly between different replicas, and the impact of node failures and network jitter must also be considered. How to achieve efficient data synchronization while ensuring data consistency and minimizing the impact of synchronization on system performance is the problem that this application's embodiments aim to solve.

[0112] As one implementation method, in order to efficiently achieve real-time synchronization of land and resources data across various storage servers, this application considers the issue of the order of data changes during the data synchronization process. That is, the order in which different nodes receive change requests may be inconsistent. This application ensures that the order of change operations is unified across all copies.

[0113] Specifically, this application embodiment addresses data sharding in a large-scale distributed system by employing a consistent hashing algorithm to distribute data shards evenly across different physical nodes and generating multiple replicas for each shard, thereby improving data availability and fault tolerance.

[0114] Consistent hashing maps nodes and data shards to a virtual hash ring. Each node and data shard has a hash value, and these hash values ​​are ordered on the ring. A data shard is located on the ring using its hash value, and the first node in a clockwise direction is found, at which point the data shard is assigned. To increase data availability and fault tolerance, each data shard has multiple replicas, which are also distributed to different nodes using consistent hashing.

[0115] One embodiment involves a distributed database system that uses a consistent hashing algorithm to distribute data shards across 10 server nodes. Data shard A might be assigned to node N1, and data shard B might be assigned to node N5. To improve reliability, data shard A may also have replicas on nodes N2 and N4, and data shard B may have replicas on nodes N6 and N8.

[0116] Efficient data synchronization and consistency maintenance comprehensively consider multiple aspects such as data sharding, replica management, data change propagation, consensus algorithms, and network optimization. By using consistent hashing algorithms for data sharding and replica management, incremental synchronization and consensus algorithms to ensure data consistency, and optimizing the network and handling faults, efficient, reliable, and orderly data synchronization can be achieved in large-scale distributed systems.

[0117] As one implementation method, it also includes:

[0118] Obtain vector clock information and change information of several nodes. Mark the change information with timestamps according to the vector clock information to obtain change record information used to record the order of change.

[0119] Specifically, in the data synchronization process, this application introduces a vector clock mechanism to assign a globally unique timestamp to each data change operation, which is used to record the order of changes. Simultaneously, an efficient data synchronization protocol is designed, using incremental synchronization and compression techniques to reduce the amount of data transmitted over the network and improve synchronization efficiency.

[0120] Each node maintains a vector clock to record changes to itself and other nodes. For example, suppose there are three nodes A, B, and C, each with a vector clock. Initially, these vector clocks are all [0,0,0].

[0121] When node A makes a data change, such as updating the attributes of a plot of land, node A increments its own vector clock count, for example, from [0,0,0] to [1,0,0], and records the details of this change.

[0122] Node A sends the change information and the updated vector clock to other nodes B and C. After receiving it, node B updates its own vector clock according to node A's vector clock, for example, from [0,0,0] to [1,0,0], and records its update status in the system.

[0123] Nodes B and C combine change information with their corresponding vector clocks to determine the order in which changes occur. For example, if node B's vector clock is [1,0,0] and node C's vector clock is [0,1,0], then a change at node B is considered to have occurred after node A but before node C.

[0124] During synchronization, only the incremental part of the changes, that is, the changes since the last synchronization, is transmitted, and compression algorithms are used to reduce the amount of data, thereby improving the efficiency of network transmission.

[0125] In one embodiment, node A updates the attribute of data item X, while node B updates the attribute of data item Y. Node A's vector clock is updated to [1,0,0], and node B's vector clock is updated to [0,1,0]. During synchronization, both nodes A and B send their change information and corresponding vector clocks to node C. After receiving this information, node C merges these changes and determines the order of changes based on the vector clocks, thereby ensuring data consistency across all nodes.

[0126] As one implementation method, it also includes:

[0127] Obtain node monitoring status information. If the node monitoring status information indicates a node failure, obtain the hot and cold data of the backup copy and restore the hot and cold data of the backup copy to the hierarchical cache corresponding to the node failure.

[0128] Specifically, this application's embodiments design a robust fault detection and recovery mechanism to address abnormal situations such as node failures and network jitter. A heartbeat mechanism monitors the node's status in real time; if a node failure is detected, a fault recovery process is triggered, retrieving the latest data from other replica nodes to ensure data availability. Simultaneously, retry and timeout mechanisms are introduced to address synchronization failures caused by network jitter.

[0129] The system monitors node status in real time through a heartbeat mechanism. A heartbeat signal is a message sent periodically by a node to notify the system that it is functioning normally. If a node fails to respond to a heartbeat within a predetermined time, the system will determine that the node may have malfunctioned.

[0130] For example, suppose there are 10 nodes in the system, and each node sends a heartbeat signal to the monitoring system every 10 seconds. If the monitoring system does not receive a heartbeat signal from a node within 30 seconds, the system will mark that node as a faulty node and trigger the fault recovery process.

[0131] In addition to the heartbeat mechanism, the system can also detect node failures or performance degradation through other means, such as load balancing and abnormal response times.

[0132] When a node failure is detected, the system will automatically trigger a fault recovery process. The general process includes obtaining the latest data from the backup replica node and restoring it to the tiered cache corresponding to the failed node.

[0133] For example, if node A fails, nodes B and C store node A's primary data and backup data, respectively. The system will extract node A's hot and cold data from nodes B and C and restore them to the corresponding cache of node A.

[0134] When restoring data from backup copies, the system will prioritize restoring hot data, as this data has the greatest impact on business operations. The restoration of cold data can be carried out at a later time when the system is in good condition, within a preset maximum time.

[0135] For example, if node A cached user's geographic information data before the failure, hot data might be geographic information that the user frequently accessed in the past few days, while cold data might be historical data. The system will first restore the data from the most recent days, and then restore the earlier data.

[0136] When network jitter causes data synchronization failure, the system automatically retryes the synchronization operation so that the data can eventually be correctly synchronized to the faulty node.

[0137] For example, if network jitter causes synchronization failure when data is transmitted from node A to node B, the system will retry several times, such as 5 times, with a certain interval set between each retry.

[0138] The system is configured with a reasonable timeout period to reduce indefinite waiting for network recovery. After the timeout, the system will report the problem and attempt other recovery methods.

[0139] For example, if the recovery process of node A times out due to network issues, and the timeout period is set to 60 seconds, the system will record the event and attempt to recover from other backup copies.

[0140] The data storage method enables the system to quickly recover data and maintain high availability when nodes fail by monitoring node status in real time, triggering automatic recovery processes, and handling network jitter.

[0141] To improve data consistency among all replicas of the same shard, a distributed consensus algorithm based on the Raft protocol is designed. A primary replica is elected and is responsible for coordinating data synchronization and replication. When data changes, the change operation is first executed on the primary replica, and then the changes are synchronized to other replicas, eventually ensuring that the data among the replicas reaches a consistent state.

[0142] As one implementation method, it also includes:

[0143] Generate a transaction ID for each change information in the same change record to obtain a unique transaction ID corresponding to the change information.

[0144] Specifically, this application addresses the issue of the sequential nature of data changes by introducing a global transaction mechanism. A unique transaction ID is generated for each transaction, and this transaction ID is embedded into the data change operation. During data synchronization, the execution order of the change operations is determined based on the order of the transaction IDs, ensuring consistency in the order of data changes across different nodes.

[0145] A unique transaction ID is generated for each data change operation using globally incrementing sequence numbers, timestamps, and UUIDs. The uniqueness of the transaction ID ensures that each change operation can be uniquely identified. Within each data change operation, the generated transaction ID is embedded in the change record. When data is synchronized across different nodes, the order of change operations can be sorted using the transaction ID. During data synchronization, nodes determine the execution order of data changes based on the size and order of the transaction IDs, ensuring consistency in the data change order across all nodes.

[0146] Each time a data change operation occurs, the system generates a unique transaction ID. For example, globally incrementing sequence numbers like TID001 and TID002 are used to ensure the uniqueness and order of each transaction ID. The transaction ID is added to the data change record. When node A synchronizes data to node B, node B sorts the change records according to the transaction ID, executing the change operation for TID001 first, then the change operation for TID002. This ensures that the data change order on node B is consistent with that on node A.

[0147] By introducing a global transaction mechanism and generating a unique transaction ID for each change operation, the order of data change operations can be effectively managed and synchronized. The use of transaction IDs not only helps identify each change operation but also ensures the sequential consistency of change operations in a distributed system, thereby avoiding data problems caused by inconsistencies in order.

[0148] Furthermore, to minimize the impact of data synchronization on system performance, asynchronous replication is employed for data synchronization. Synchronization operations are made asynchronous, with change operations cached through a message queue, allowing the primary replica to respond to clients immediately after completing changes without waiting for synchronization to finish. Simultaneously, an efficient batch synchronization mechanism is designed to merge multiple change operations for synchronization, reducing synchronization frequency and improving system throughput.

[0149] Asynchronous replication ensures that data changes are not immediately synchronized to replica nodes, but rather performed asynchronously. The master node responds to clients immediately after completing the change, without waiting for replica nodes to synchronize. Message queue caching caches data change operations in a message queue, which handles data synchronization in the background, sending the change operations from the master node to the replica nodes. A batch synchronization mechanism combines multiple change operations into a single batch for synchronization. By reducing the synchronization frequency—that is, not synchronizing every change immediately—it improves system throughput and efficiency. This reduces the impact of synchronization operations on system performance, improving system response speed and processing capacity.

[0150] Reference Figure 2 and Figure 3 This application provides a data storage system for land and resources planning, the system comprising:

[0151] The information acquisition module is used to acquire specified land and resources storage data, perform data segmentation processing on the stored data, and obtain data segmentation information.

[0152] The sharding analysis module is used to perform real-time analysis of data sharding information to obtain hot data and cold data. Hot data is synchronously copied, while cold data is asynchronously copied.

[0153] The partitioning module is used to obtain memory status information, partition the memory status information to obtain hierarchical information, adjust the hierarchical information according to hot data, and store hot data into several hierarchical caches.

[0154] The dynamic adjustment module is used to acquire data access traffic information in real time, and dynamically adjust the data sharding information based on the data access traffic information to obtain distribution update information for updating hot data and cold data.

[0155] Specifically, as one implementation method, the specific steps of the sharding analysis module in synchronously replicating hot data are as follows:

[0156] Obtain replication factor information and hotspot synchronization frequency information, replicate hotspot data several times based on replication factor information, and adjust the replication rate of hotspot data based on hotspot synchronization frequency information.

[0157] In one implementation method, the specific steps of the sharding analysis module for asynchronously copying cold data are as follows:

[0158] Obtain replication factor information and cold synchronization frequency information, replicate cold data a number of times based on replication factor information, and adjust the replication rate of cold data based on cold synchronization frequency information.

[0159] As one embodiment, it further includes:

[0160] The status analysis module acquires operational indicator information in real time, analyzes the operational indicator information, and obtains operational status information.

[0161] If the running status information indicates an abnormal mode, a backup copy is built based on the hierarchical cache, and hot data and cold data are transferred and stored to the backup copy.

[0162] As one embodiment, it further includes:

[0163] The replica adjustment module is used to dynamically adjust the number of backup replicas based on data access traffic information and operation indicator information. Specifically, when the operation status information indicates an abnormal mode, the number of backup replicas for the current hot data is increased, and when the operation status information indicates a normal mode, the number of backup replicas for the current cold data is decreased.

[0164] As one embodiment, it further includes:

[0165] The clock signal acquisition module is used to acquire vector clock information and change information of several nodes. Based on the vector clock information, the change information is timestamped to obtain change record information for recording the order of events.

[0166] As one embodiment, it further includes:

[0167] The node monitoring module is used to obtain node monitoring status information. When the node monitoring status information indicates a node failure, it obtains the hot and cold data of the backup copy and restores the hot and cold data of the backup copy to the hierarchical cache corresponding to the node failure.

[0168] As one embodiment, it further includes:

[0169] The global transaction module generates a transaction ID for each change in the same change record, obtaining a unique transaction ID corresponding to the change information.

[0170] As one embodiment, it further includes:

[0171] The status analysis module is also used to analyze operational metrics including CPU utilization, memory usage, disk information, and network traffic.

[0172] Reference Figure 4 This application provides a data storage device for land and resources planning, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to execute the data storage method for land and resources planning as described above.

[0173] This application provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the data storage method for land and resources planning as described above when it is run.

[0174] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device and product described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed methods, systems, apparatus and program products can be implemented in other ways.

[0176] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0177] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data storage method for land and resources planning, characterized in that, include: Land and resources data are segmented to obtain data segmentation information; Real-time analysis of data sharding information is performed to obtain hot data and cold data. The analysis and judgment of hot data and cold data are based on whether the access frequency and importance of the data meet the classification criteria. Hot data is synchronously replicated using a distributed consensus algorithm based on the Paxos protocol, and cold data is asynchronously replicated using a distributed log synchronization mechanism based on the Raft protocol. Memory status information is obtained and divided into hierarchical information, which includes three levels: local cache, distributed cache and persistent storage. The hierarchical information is adjusted according to hot data and hot data is stored in several hierarchical caches. Real-time data access traffic information is acquired, and data sharding information is dynamically adjusted based on the data access traffic information. Specifically, the load balancing algorithm and cache management strategy are used to dynamically adjust the distribution update information for updating hot data and cold data. Real-time acquisition of operational metrics information, analysis of operational metrics information to obtain operational status information; when the operational status information indicates an abnormal mode, a backup copy is built based on tiered caching, and hot data and cold data are transferred and stored to the backup copy; among which, operational metrics information includes CPU utilization information, memory usage information, disk information, and network traffic information; Obtain vector clock information and change information of several nodes, and timestamp the change information according to the vector clock information to obtain change record information used to record the order of change. Obtain node monitoring status information. If the node monitoring status information indicates a node failure, obtain the hot and cold data of the backup copy and restore the hot and cold data of the backup copy to the hierarchical cache corresponding to the node failure.

2. The data storage method for land and resources planning according to claim 1, characterized in that, The specific steps for synchronously replicating hot data are as follows: Obtain replication factor information and hotspot synchronization frequency information, replicate hotspot data several times based on replication factor information, and adjust the replication rate of hotspot data based on hotspot synchronization frequency information.

3. The data storage method for land and resources planning according to claim 1, characterized in that, The specific steps for asynchronously copying cold data are as follows: Obtain replication factor information and cold synchronization frequency information, replicate cold data a number of times based on replication factor information, and adjust the replication rate of cold data based on cold synchronization frequency information.

4. The data storage method for land and resources planning according to claim 1, characterized in that, Also includes: The number of backup copies is dynamically adjusted based on data access traffic information and operational metrics. Specifically, if the operational status information indicates an abnormal mode, the number of backup copies of the current hot data is increased, and if the operational status information indicates a normal mode, the number of backup copies of the current cold data is decreased.

5. A data storage system for land and resources planning, characterized in that, include: The information acquisition module is used to acquire specified land and resources storage data, perform data segmentation processing on the stored data, and obtain data segmentation information. The sharding analysis module is used to perform real-time analysis of data sharding information to obtain hot data and cold data. Hot data is synchronously copied, while cold data is asynchronously copied. The partitioning module is used to obtain memory status information, partition the memory status information to obtain hierarchical information, adjust the hierarchical information according to hot data, and store hot data into several hierarchical caches. The dynamic adjustment module is used to acquire data access traffic information in real time and dynamically adjust the data sharding information based on the data access traffic information to obtain distribution update information for updating hot data and cold data. The status analysis module acquires operational indicator information in real time, analyzes the operational indicator information, and obtains operational status information. If the running status information indicates an abnormal mode, a backup copy is built based on the tiered cache, and hot data and cold data are transferred and stored to the backup copy; the running indicator information includes CPU utilization information, memory usage information, disk information and network traffic information; The clock signal acquisition module is used to acquire vector clock information and change information of several nodes. Based on the vector clock information, the change information is timestamped to obtain change record information for recording the order of occurrence. The node monitoring module is used to obtain node monitoring status information. When the node monitoring status information indicates a node failure, it obtains the hot and cold data of the backup copy and restores the hot and cold data of the backup copy to the hierarchical cache corresponding to the node failure.

6. A data storage device for land and resources planning, characterized in that, It includes a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the data storage method for land and resources planning as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the data storage method for land and resources planning as described in any one of claims 1-4 when it is run.

Images