Data storage method, system and equipment for territorial resource planning and medium

Abstract

The invention relates to the technical field of territorial resource data processing, in particular to a data storage method, system and device for territorial resource planning and a medium, and the method comprises the steps: carrying out the fragmentation processing of territorial resource data, and obtaining data fragmentation information; performing real-time analysis on the data fragment information to obtain hot data and cold data, performing synchronous replication processing on the hot data, and performing asynchronous replication processing on the cold data; obtaining memory state information, dividing the memory state information to obtain grading information, adjusting the grading information according to the hotspot data, and storing the hotspot data to a plurality of grading caches; the data access flow information is acquired in real time, the data fragmentation information is dynamically adjusted according to the data access flow information, and the distribution updating information used for updating the hot data and the cold data is obtained, so that territorial resources can be stably stored, the data consistency is improved, and the system reliability and the user experience are improved.

Description

Technical Field

[0001] The present application relates to the technical field of land and resources data processing, and in particular to a data storage method, system, device and medium for land and resources planning. Background Art

[0002] Land and resources information plays a vital role in modern society, involving land use, resource exploration, environmental protection, urban and rural planning and other aspects. With the rapid development of the economy and the acceleration of urbanization, the complexity of land and resources management and the amount of data are increasing, and the requirements for information storage and management are also increasing. This information requires not only efficient storage, but also flexible data processing capabilities to match. In traditional data storage methods, data is usually stored in a single database or storage system. Although this centralized storage method simplifies data management to a certain extent, as the amount of data increases, the single storage system faces performance bottlenecks and scalability problems. In order to cope with these existing problems, a distributed storage system is used to store land information.

[0003] Distributed storage systems not only improve the capacity and processing power of storage systems by distributing data to multiple nodes, but also improve the availability and reliability of data. In such a system, data is divided into multiple fragments, each of which is stored on a different node. In this way, the system can expand storage capacity and computing power by adding nodes. However, while achieving its advantages, distributed storage systems also have problems. Among them, one of the most significant problems is data consistency. The data consistency problem stems from how to ensure that the data on all nodes is eventually consistent when multiple nodes operate on the same data at the same time. In a distributed system, since data is stored on multiple nodes, network delays and failures between nodes may cause differences between copies of data, thus causing data consistency problems.

[0004] When multiple nodes modify data at the same time, data consistency cannot be guaranteed, resulting in inconsistent data copies, incorrect query results, and conflicts and data loss when the system handles concurrent operations. The data obtained by users does not match the actual data, which affects the accuracy of decision-making and further affects the reliability of the system and user experience. The above problems need to be solved. Summary of the invention

[0005] In order to enable stable storage of land resources and improve data consistency, thereby improving system reliability and user experience, this application provides a data storage method, system, device and medium for land resources planning, which adopts the following technical solutions:

[0006] In a first aspect, the present application provides a data storage method for land and resources planning, comprising:

[0007] Slice the land and resources data to obtain data slicing information;

[0008] Perform real-time analysis on data sharding information to obtain hot data and cold data, perform synchronous replication on hot data, and perform asynchronous replication on cold data;

[0009] Acquire memory status information, divide the memory status information to obtain hierarchical information, adjust the hierarchical information according to hot data, and store the hot data in a plurality of hierarchical caches;

[0010] The data access traffic information is obtained in real time, and the data sharding information is dynamically adjusted according to the data access traffic information to obtain the distribution update information for updating the hot data and the cold data.

[0011] Preferably, the specific steps of synchronously replicating the hotspot data are:

[0012] The replication factor information and the hotspot synchronization frequency information are obtained, the hotspot data is replicated several times according to the replication factor information, and the replication rate of the hotspot data is adjusted according to the hotspot synchronization frequency information.

[0013] Preferably, the specific steps of performing asynchronous replication processing on cold data are:

[0014] Acquire replication factor information and cold synchronization frequency information, replicate cold data several times according to the replication factor information, and adjust the replication rate of cold data according to the cold synchronization frequency information.

[0015] Preferably, it also includes:

[0016] Obtain operation index information in real time, analyze the operation index information, and obtain operation status information;

[0017] When the operation status information is judged to be in an abnormal mode, a backup copy is constructed based on the hierarchical cache, and the hot data and cold data are transferred and stored in the backup copy.

[0018] Preferably, it also includes:

[0019] The number of backup copies is dynamically adjusted based on data access traffic information and operating indicator information. When the operating status information is judged to be an abnormal mode, the number of backup copies of the current hot data is increased. When the operating status information is judged to be a normal mode, the number of backup copies of the current cold data is reduced.

[0020] Preferably, it also includes:

[0021] Vector clock information and change information of several nodes are obtained, and the change information is timestamped according to the vector clock information to obtain change record information for recording the sequence.

[0022] Preferably, it also includes:

[0023] Obtain node monitoring status information, and when the node monitoring status information indicates a node failure, obtain hot data and cold data of the backup copy, and restore the hot data and cold data of the backup copy to the hierarchical cache corresponding to the node failure.

[0024] In a second aspect, the present application provides a data storage system for land and resources planning, comprising:

[0025] The information acquisition module is used to obtain the specified land and resources storage data, perform sharding on the stored data, and obtain data sharding information;

[0026] The shard analysis module is used to analyze the data shard information in real time, obtain hot data and cold data, perform synchronous replication on the hot data, and perform asynchronous replication on the cold data;

[0027] A partitioning module is used to obtain memory status information, partition the memory status information to obtain hierarchical information, adjust the hierarchical information according to hot data, and store the hot data in a plurality of hierarchical caches;

[0028] The dynamic adjustment module is used to obtain data access traffic information in real time, dynamically adjust the data sharding information according to the data access traffic information, and obtain distribution update information for updating hot data and cold data.

[0029] In a third aspect, the present application provides a data storage device for land and resources planning, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the data storage method for land and resources planning as described above.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the data storage method for land and resources planning as described above when running.

[0031] In summary, compared with the prior art, the technical solution provided by this application has at least the following beneficial effects:

[0032] The present application processes the storage data of the designated land and resources, analyzes the data sharding information therein to obtain hot data and cold data, adopts strongly consistent synchronous replication processing for the hot data, and adopts eventually consistent asynchronous replication processing for the cold data. After processing, the memory status information is obtained and divided to obtain hierarchical information divided into several levels, and the hierarchical information is adjusted through the hot data to store the hot data in several hierarchical caches, and then the data access traffic information is obtained in real time, and the data sharding information is dynamically adjusted according to the data access traffic information to obtain the distribution update information for updating the hot data and the cold data, and the data sharding distribution strategy is adjusted in real time through the distribution update information, so that the land and resources can be stored stably, and the consistency of the data is improved, thereby improving the reliability of the system and the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of a data storage method for land resources planning described in an embodiment of the present application.

[0034] Figure 2 It is a module diagram of a data storage system for land and resources planning described in an embodiment of the present application.

[0035] Figure 3 It is a module schematic diagram of the data storage system described in an embodiment of the present application.

[0036] Figure 4 It is a structural schematic diagram of a data storage device for land resources planning described in an embodiment of the present application.

[0037] Description of reference numerals:

[0038] 1. Information acquisition module; 2. Sharding analysis module; 3. Partitioning module; 4. Dynamic adjustment module; 5. Status analysis module; 6. Replica adjustment module; 7. Clock signal acquisition module; 8. Node monitoring module; 9. Global transaction module. DETAILED DESCRIPTION

[0039] The following combination Figure 1-Figure 4 The present application is described in further detail. The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting.

[0040] Reference Figure 1 The data storage method of land resources planning involved in this application specifically includes:

[0041] Step S1: Slice the land and resources data to obtain data slicing information;

[0042] Step S2: Analyze the data sharding information in real time to obtain hot data and cold data, perform synchronous replication on the hot data, and perform asynchronous replication on the cold data;

[0043] Step S3: acquiring memory status information, dividing the memory status information to obtain hierarchical information, adjusting the hierarchical information according to hotspot data, and storing the hotspot data in a plurality of hierarchical caches;

[0044] Step S4: acquiring data access flow information in real time, dynamically adjusting data sharding information according to the data access flow information, and obtaining distribution update information for updating hot data and cold data.

[0045] Specifically, the present application can dynamically adjust the level of data consistency according to different business scenarios and needs by designing a flexible data consistency solution. Obtain scenario feature information, that is, preset conditions, obtain corresponding land and resources storage data according to scenario characteristics for selection, and pre-process the storage data, and then extract a representative sample from the massive data for preliminary analysis and processing. By processing the storage data of land and resources, the data sharding information therein is analyzed to obtain hot data and cold data, and the hot data is synchronously replicated with strong consistency, and the cold data is asynchronously replicated with final consistency. After processing, the memory status information is obtained and divided to obtain hierarchical information divided into several levels, and the hierarchical information is adjusted through the hot data, so that the hot data is stored in several hierarchical caches, and then the data access traffic information is obtained in real time, and the data sharding information is dynamically adjusted according to the data access traffic information, and the distribution update information for updating the hot data and the cold data is obtained. The distribution update information is used to adjust the data sharding distribution strategy in real time, so that the land and resources can be stably stored, and the consistency of the data is improved, thereby improving the reliability of the system and the user experience.

[0046] In large-scale distributed systems, data consistency is an issue that needs to be taken seriously, especially when performing data sharding and storage. The hot data in the embodiments of the present application is data that is frequently accessed and modified. In a distributed system, hot data is the focus of user requests or data with high frequency operations. Cold data refers to data with low access frequency and few modification operations. These data are historical data, archived data, or secondary data that is not frequently accessed.

[0047] The analysis and judgment of hot data and cold data is based on whether the access frequency and importance of the data meet the classification criteria.

[0048] Due to frequent access and modification operations, hot data usually encounters high-concurrency requests. Frequent changes in data mean that any delay or inconsistency may have a serious impact on user experience or business logic, so high consistency requirements are placed on hot data.

[0049] The embodiment of the present application extracts metadata of land and resources data from a storage system. The metadata includes data shard information, such as shard location, size, etc. The data shards are preprocessed to extract useful information and format them. Real-time analysis tools such as Apache Kafka, Apache Flink, or Spark Streaming are used to identify hot data, where hot data refers to data with high access frequency or importance, and an access rate threshold or importance threshold is specifically set for identification.

[0050] The synchronous replication process adopted in the embodiment of the present application is a strong consistency strategy. For hot data, a strong consistency strategy needs to be adopted to ensure that the data on all nodes are always consistent. For example, a distributed consensus algorithm based on the Paxos protocol is used to process high-concurrency write requests. The Paxos protocol improves data consistency by majority voting, so that data consistency can be maintained even when a node fails.

[0051] Cold data is data that is accessed less frequently and is identified using the same analytical methods as hot data.

[0052] The access frequency of cold data is much lower than that of hot data, which means that this data is not frequently read or modified. Due to the low access frequency, the concurrent operations of cold data are relatively few. Therefore, the consistency requirements for cold data are relatively loose, and the final consistency is usually sufficient to meet business needs, so that more device resources can be inclined to the processing and allocation of hot data.

[0053] For cold data, an eventual consistency strategy is used to handle data synchronization. Eventual consistency allows data to be inconsistent for a short period of time, but it will eventually reach a consistent state. For example, a distributed log synchronization mechanism based on the Raft protocol is used to synchronize data between nodes by recording logs and playing them sequentially. Using the characteristics of the protocol itself, the Paxos protocol is used to handle high-concurrency write scenarios, while the Raft protocol is suitable for data synchronization with low-frequency updates, which can allocate resources more reasonably, thereby improving the system's user experience.

[0054] As one implementation method, the specific steps of synchronously replicating hotspot data are as follows:

[0055] The replication factor information and the hotspot synchronization frequency information are obtained, the hotspot data is replicated several times according to the replication factor information, and the replication rate of the hotspot data is adjusted according to the hotspot synchronization frequency information.

[0056] Specifically, the replication factor information is the number of copies of the data in the system. For example, if the replication factor is 3, then each piece of data will have 3 copies stored on different nodes.

[0057] The system obtains the current replication factor setting from configuration management or metadata storage. The replication factor is adjusted based on the importance and access frequency of the data. Hot data needs to be set with a higher replication factor to enable high availability and fast access.

[0058] The hotspot synchronization frequency information is the time interval for updating the hotspot data copy. For example, if the synchronization frequency is once a minute, the system will analyze the data shard information every minute, thereby playing the role of updating the copy of all hotspot data every minute.

[0059] The system obtains synchronization frequency settings from configuration management or business rules. The synchronization frequency is adjusted according to the needs of data updates to balance consistency and performance requirements.

[0060] Hotspot data is replicated from the primary storage node to multiple backup nodes according to the replication factor. The replication process usually includes data transfer and write operations so that all copies are consistent. For example, if the replication factor of hotspot data is set to 3, the system will replicate the data from the primary node to 2 other nodes.

[0061] According to the hotspot synchronization frequency, adjust the rate of the replication process. For example, if the synchronization frequency is once a minute, the system may need to complete the replication and synchronization of data within every minute. The system can use flow control and load balancing technology to optimize the replication rate so that the stability of data synchronization can be maintained even under high load conditions.

[0062] In one embodiment, the hotspot data in a GIS system includes the latest geological survey results and environmental monitoring data. In order to enable users to obtain the latest geological data when performing real-time queries, the system needs to perform efficient synchronous replication of these data;

[0063] Set the replication factor of hot data to 5, so that each hot data has a copy on 5 different nodes;

[0064] The synchronization frequency is set to every 10 minutes, because geological data does not change frequently but needs to maintain high consistency;

[0065] After receiving the new geological data update, the master node distributes the data to the five nodes;

[0066] Every 10 minutes, the system checks and synchronizes data copies on all nodes to improve consistency;

[0067] If the system detects that some nodes are under high load, the replication process will automatically adjust the rate to avoid overload.

[0068] In another embodiment, in an urban planning system, hot data includes the latest building permit applications and urban development dynamics. These data need to be synchronized quickly to support real-time planning decisions;

[0069] Set the replication factor of hot data to 3 so that the data is backed up on three nodes;

[0070] Set the synchronization frequency to every 5 minutes to keep the data up to date in a rapidly changing urban development environment;

[0071] After receiving the new license application, the primary storage node copies the data to the three backup nodes;

[0072] The system checks data consistency every 5 minutes and performs synchronization operations when necessary;

[0073] If insufficient network bandwidth is detected, the system will adjust the data replication rate to optimize performance and ensure the stability of data synchronization.

[0074] The above steps illustrate how to efficiently manage and synchronize hotspot data by configuring the replication factor and synchronization frequency. This approach can improve the high availability and consistency of hotspot data, while dynamically adjusting the data replication and synchronization strategy based on actual needs.

[0075] As one implementation method, the specific steps of asynchronously replicating cold data are as follows:

[0076] Acquire replication factor information and cold synchronization frequency information, replicate cold data several times according to the replication factor information, and adjust the replication rate of cold data according to the cold synchronization frequency information.

[0077] Specifically, the replication factor information in the embodiment of the present application is the same information as that obtained when processing hot data, but the replication factor information of hot data and cold data is not limited to the same value.

[0078] The cold synchronization frequency information is the synchronization update frequency of cold data, which determines the synchronization interval between replicas. For example, if the cold synchronization frequency is once a day, it means synchronization once every 24 hours.

[0079] According to the preset replication factor, cold data is replicated to multiple storage locations, which involves creating multiple copies to be stored on different storage devices or nodes. For example, if the replication factor is 3, the cold data is replicated on three different storage devices.

[0080] Adjust the data replication rate and frequency based on the preset synchronization frequency information. For example, if the synchronization frequency is once a day, the system will set the data replication operation to be performed daily and during off-peak hours to reduce the impact on system performance.

[0081] In one embodiment, the replication factor is set to 3, which means that each cold data will be replicated to 3 different storage locations. The cold synchronization frequency is set to once a day, and the cold data copies are synchronized and updated once every 24 hours;

[0082] Assume that the cold data is "geological survey data", which needs to be replicated three times. The system replicates the original data to three different hard disks or storage nodes to ensure that the data has sufficient high availability and redundancy;

[0083] According to the cold synchronization frequency, the system automatically performs cold data synchronization at a specific time every day, such as 2 a.m. In order to reduce the impact on system performance during peak hours, the replication process is performed during non-peak hours, and appropriate bandwidth control strategies are used to optimize the replication process;

[0084] The system will continuously monitor the replication status and synchronization of cold data. If any replica synchronization is delayed or fails, the system will automatically trigger an alarm and take measures, such as retrying replication or re-adjusting the replication rate. Through the above steps, the system can ensure reliable storage and efficient access to cold data while reducing performance issues caused by frequent synchronization.

[0085] As one implementation method, memory status information is acquired, the memory status information is divided to obtain hierarchical information, the hierarchical information is adjusted according to hot data, and the hot data is stored in a plurality of hierarchical caches.

[0086] Specifically, in a large-scale distributed system, the efficiency of data access and the response speed of the system are often limited by the delay of data storage and the access speed. The embodiments of the present application provide an efficient cache architecture that can significantly improve the performance of the system.

[0087] The memory status information related to the storage is obtained, and the memory status information is divided to obtain hierarchical information. The hierarchical information includes three levels: local cache, distributed cache, and persistent storage. The corresponding external hardware is regulated according to the corresponding information.

[0088] Local cache refers to the cache maintained on each computing node or server, which is used to store copies of frequently accessed data on the node. It is stored in memory and interacts directly with the application process or operating system. Because the data is directly stored in the local memory, the data access latency is extremely low, which reduces the access to remote nodes and reduces the consumption of network bandwidth.

[0089] Distributed cache is a cache system across multiple computing nodes, consisting of multiple cache nodes. It is used to share cache data throughout the system to improve the availability and consistency of data access. Specifically, the access load is balanced by distributing cache data among multiple cache nodes. Among them, the distributed cache system has a redundancy mechanism. When a node fails, other nodes can continue to provide services. Specifically, cache nodes are added as needed to handle larger data volumes and higher access loads.

[0090] Persistent storage refers to storing data in a stable storage medium, such as a hard disk, SSD, or database system. It is used to save data for long-term storage and recovery. Data will not be lost due to node failure or restart, which improves the long-term reliability of data.

[0091] As one implementation method, data access traffic information is acquired in real time, and data sharding information is dynamically adjusted according to the data access traffic information to obtain distribution update information for updating hot data and cold data.

[0092] Specifically, the embodiment of the present application monitors data access traffic in real time and adjusts the data sharding distribution strategy according to the prediction results. Specifically, the load balancing algorithm and cache management strategy are dynamically adjusted to reasonably distribute hot data and optimize cache utilization and data consistency.

[0093] Among them, this application uses a real-time monitoring tool to track the access traffic of each data fragment, including read and write operations. The collected data traffic information is processed to identify hot data, that is, high access frequency, and cold data, that is, low access frequency. The analysis results should include the trend and change pattern of data access volume. Based on the results of traffic analysis, the data sharding strategy is dynamically adjusted. For hot data, increase data copies and distribute them to more nodes to reduce the load; for cold data, reduce the number of copies and optimize storage resources. Implement the sharding redistribution mechanism in the distributed system and apply the new sharding configuration to the system. In the process of data sharding adjustment, the consistency and availability of data are guaranteed.

[0094] After monitoring the system performance after the sharding adjustment, evaluate the processing effect of hot and cold data. Continuously optimize and adjust the strategy based on the feedback data to improve the overall performance and data consistency of the system.

[0095] As one implementation mode, it also includes:

[0096] Obtain operation index information in real time, analyze the operation index information, and obtain operation status information;

[0097] When the operation status information is judged to be in an abnormal mode, a backup copy is constructed based on the hierarchical cache, and the hot data and cold data are transferred and stored in the backup copy.

[0098] The operation indicator information includes CPU usage information, memory usage information, disk information and network traffic information.

[0099] Specifically, the embodiments of the present application improve the fault tolerance of the system through mechanisms such as data redundancy and failover to cope with various abnormal situations and further improve the consistency of system data. The operating indicator information includes real-time acquisition of the device's CPU usage, memory usage, disk I / O, and network traffic light information.

[0100] Through time series analysis and clustering algorithms, the real-time acquisition of equipment CPU usage, memory usage, disk I / O, network traffic light information and other operating indicators are analyzed to collect operating status information. Based on the operating status information, the current state is judged to be normal or abnormal. In the case of abnormal mode, data migration and replica creation operations are triggered to work in conjunction with the aforementioned data distribution strategy.

[0101] When the system detects an abnormal pattern, it triggers a data migration operation, specifically moving hot and cold data to backup copies to reduce the load on the primary system and protect the data.

[0102] The system dynamically adjusts the number of copies based on the importance of the data, access frequency, and current load. Under normal circumstances, the number of copies of cold data is reduced to save resources; when a fault warning is issued, the number of copies of key data is increased to ensure data security and availability.

[0103] As one of the implementation methods, the number of backup copies is dynamically adjusted according to data access traffic information and operation indicator information. When the operation status information is judged to be an abnormal mode, the number of backup copies of the current hot data is increased, and when the operation status information is judged to be a normal mode, the number of backup copies of the current cold data is reduced.

[0104] Specifically, in order to optimize system performance and data availability, the system implements a strategy for dynamically adjusting the number of backup copies in the embodiment of the present application. The strategy adjusts the number of copies of different data shards based on data access flow information, operating indicator information, and system abnormality. This strategy improves the overall performance, load balancing capability, concurrent processing capability, fault tolerance, and data consistency of the system.

[0105] The system dynamically adjusts the number of replicas for different data shards. This adjustment strategy takes into account the importance of the data, the access frequency, and the current system load.

[0106] As hot data is frequently accessed, the number of its copies increases, which improves availability during high-concurrency access. The number of cold data copies decreases, reducing storage and synchronization overhead, thereby optimizing system performance.

[0107] The embodiment of the present application adjusts the number of copies in consideration of data importance and access frequency. For example, transaction records and user data will be marked as critical data, so that the number of copies will be increased first to improve the high availability of data and the fault tolerance of the system.

[0108] In addition, for data with high access frequency, more copies will be added to meet high concurrency requirements; for data with low access frequency, fewer copies will be added to save storage and network resources.

[0109] Furthermore, when the system is running normally, the number of replicas is adjusted according to the load to balance storage resources and data synchronization overhead. In high load conditions, the number of replicas of important data is increased, while in low load conditions, the number of replicas of cold data is reduced.

[0110] The embodiments of the present application can improve system performance by reducing the number of copies of cold data, reducing data synchronization and storage overhead, thereby improving the overall performance of the system. Dynamically adjusting the number of copies can effectively disperse the load, optimize the use of system resources, and balance the load. By increasing the number of copies of hot data, the system's ability to handle high concurrent access is improved. Increasing the number of copies of key data enhances the system's ability to resist failures. The number of copies can also be reasonably adjusted to ensure consistency and reliability of data between different copies. By dynamically adjusting the number of copies, data can always be quickly accessed and reliably stored under different loads and operating conditions.

[0111] There are a large number of data shards in the system, which are distributed on different physical nodes, and each shard has multiple copies. When the data of a shard changes, it is necessary to quickly and accurately synchronize the changes to other copies, and ensure that all copies of the same shard eventually reach a consistent state. However, in a complex network environment, the data transmission delay between different copies varies greatly, and the impact of factors such as node failures and network jitter must also be considered. How to achieve efficient data synchronization while ensuring data consistency and minimize the impact of synchronization on system performance is a problem that needs to be solved in the embodiments of the present application.

[0112] As one of the implementation methods, in order to efficiently realize real-time synchronization of land and resources data in various storage servers, this application considers the sequentiality of data changes during the data synchronization process, that is, the order in which different nodes receive change requests may be inconsistent. This application unifies the order of change operations on all copies.

[0113] Specifically, the embodiments of the present application target data sharding in large-scale distributed systems, adopt a consistent hashing algorithm to allocate data shards, evenly distribute data to different physical nodes, and generate multiple copies for each shard, thereby improving data availability and fault tolerance.

[0114] The consistent hashing algorithm maps nodes and data shards to a virtual hash ring. Each node and data shard has a hash value, and these hash values ​​form an order on the ring. The data shard is located at a position on the ring by the hash value, and the first node in the clockwise direction is found and the data shard is assigned to the node. In order to increase the availability and fault tolerance of data, each data shard will be set up with multiple copies, and the copies are also assigned to different nodes through consistent hashing.

[0115] In one embodiment, a distributed database system uses a consistent hashing algorithm to distribute data shards to 10 server nodes. Data shard A may be distributed to node N1, and data shard B may be distributed to node N5. To improve reliability, data shard A will also have copies on nodes N2 and N4, and data shard B will also have copies on nodes N6 and N8.

[0116] Efficient data synchronization and consistency maintenance comprehensively considers data sharding, copy management, data change propagation, consistency algorithm, network optimization, etc. Through consistent hashing algorithm for data sharding and copy management, using incremental synchronization and consensus algorithm to ensure data consistency, as well as optimizing network and handling failures, efficient, reliable and orderly synchronization of data in large-scale distributed systems can be achieved.

[0117] As one implementation mode, it also includes:

[0118] Vector clock information and change information of several nodes are obtained, and the change information is timestamped according to the vector clock information to obtain change record information for recording the sequence.

[0119] Specifically, in the data synchronization process, the embodiment of the present application introduces a vector clock mechanism to assign a globally unique timestamp to each data change operation to record the order of the changes. At the same time, an efficient data synchronization protocol is designed to reduce the amount of data transmitted over the network and improve synchronization efficiency through incremental synchronization and compression technology.

[0120] For example, each node maintains a vector clock to record the change events of itself and other nodes. For example, there are three nodes A, B, and C, each of which has a vector clock. In the initial state, these vector clocks are all [0,0,0].

[0121] When node A makes a data change, for example, updates the attributes of a plot of land, node A increases its own vector clock count, for example, from [0,0,0] to [1,0,0], and records the detailed information of the change.

[0122] Node A sends the change information and updated vector clock to other nodes B and C. After receiving it, node B updates its own vector clock according to the vector clock of node A, for example, from [0,0,0] to [1,0,0], and records its updated status in the system.

[0123] Nodes B and C combine the change information with the corresponding vector clocks to determine the order of change. For example, if the vector clock of node B is [1,0,0] and the vector clock of node C is [0,1,0], the change of node B is considered to occur after node A but before node C.

[0124] During the synchronization process, only the incremental changes, that is, the changes since the last synchronization, are transmitted, and a compression algorithm is used to reduce the amount of data, thereby improving the efficiency of network transmission.

[0125] In one embodiment, suppose node A updates the attribute of a data item X, and node B updates the attribute of a data item Y. The vector clock of node A is updated to [1,0,0], and the vector clock of node B is updated to [0,1,0]. During the synchronization process, nodes A and B send their change information and corresponding vector clocks to node C. After receiving this information, node C merges these changes and determines the order of changes based on the vector clock, so that the data of all nodes remains consistent.

[0126] As one implementation mode, it also includes:

[0127] Obtain node monitoring status information, and when the node monitoring status information indicates a node failure, obtain hot data and cold data of the backup copy, and restore the hot data and cold data of the backup copy to the hierarchical cache corresponding to the node failure.

[0128] Specifically, the embodiments of the present application design a robust fault detection and recovery mechanism for abnormal situations such as node failure and network jitter. The heartbeat mechanism monitors the status of the node in real time. If a node failure is found, the fault recovery process is triggered to obtain the latest data from other replica nodes to ensure data availability. At the same time, a retry and timeout mechanism is introduced to deal with synchronization failures caused by network jitter.

[0129] The system monitors the status of nodes in real time through the heartbeat mechanism. The heartbeat signal is a message sent by the node periodically to inform the system that it is operating normally. If a node fails to respond to the heartbeat within the predetermined time, the system will determine that the node may have failed.

[0130] For example, suppose there are 10 nodes in the system, and each node sends a heartbeat signal to the monitoring system every 10 seconds. If the monitoring system does not receive a heartbeat signal from a node within 30 seconds, the system will mark the node as a faulty node and trigger the fault recovery process.

[0131] In addition to the heartbeat mechanism, the system can also detect node failures or performance degradation through other means, such as load balancing and response time anomalies.

[0132] When a node failure is detected, the system will automatically trigger the failure recovery process. The general process includes obtaining the latest data from the backup replica node and restoring it to the hierarchical cache corresponding to the failed node.

[0133] For example, if node A fails, node B and node C respectively store the primary data and backup data of node A. The system will extract the hot data and cold data of node A from nodes B and C and restore them to the cache corresponding to node A.

[0134] When restoring data from a backup copy, the system will prioritize hot data because these data have the greatest impact on the business. Cold data can be restored within a preset maximum time, that is, it can be restored later when the system is in good condition.

[0135] For example, before node A fails, it caches the user's geographic information data. Hot data may be the geographic information frequently accessed by users in recent days, while cold data may be historical data. The system will first restore the data of recent days, and then restore the older data.

[0136] When network jitter causes data synchronization failure, the system automatically retries the synchronization operation so that the data can eventually be correctly synchronized to the failed node.

[0137] For example, when synchronizing the data of node A to node B, network jitter occurs, resulting in synchronization failure. The system will retry several times, for example, 5 times, and set a certain interval time between each retry.

[0138] The system sets a reasonable timeout to reduce the indefinite wait for network recovery. After the timeout, the system will report the problem and try other recovery methods.

[0139] For example, if the recovery process of node A times out due to a network problem and the timeout period is set to 60 seconds, the system will record the event and try to recover from other backup copies.

[0140] The data storage method monitors node status in real time, triggers automatic recovery processes, and handles network jitter, so that the system can quickly recover data when a node fails and maintain high availability of the system.

[0141] In order to improve the data consistency between all replicas of the same shard, a distributed consistency algorithm based on the Raft protocol is designed. The master replica is elected and is responsible for coordinating data synchronization and replication. When data changes, the change operation is first performed on the master replica, and then the change is synchronized to other replicas, so that the data between the replicas eventually reaches a consistent state.

[0142] As one implementation mode, it also includes:

[0143] A transaction ID is generated for the change information in the same change record information to obtain unique transaction ID information corresponding to the change information.

[0144] Specifically, the embodiment of the present application aims at the problem of sequential data changes by introducing a global transaction mechanism, generating a unique transaction ID for each transaction, and embedding the transaction ID into the data change operation. When synchronizing data, the execution order of the change operation is determined according to the size order of the transaction ID, so that the order of data changes on different nodes is consistent.

[0145] Through globally increasing serial numbers, timestamps, UUIDs and other methods, a unique transaction ID is generated for each data change operation. The uniqueness of the transaction ID ensures that each change operation can be uniquely identified. In each data change operation, the generated transaction ID is embedded in the change record. When data is synchronized between different nodes, the order of change operations can be sorted by transaction ID. When synchronizing data, nodes determine the execution order of data changes based on the size of transaction IDs, so that the order of data changes on all nodes is consistent.

[0146] Each time a data change operation occurs, the system generates a unique transaction ID. For example, use globally incremented serial numbers TID001, TID002, etc. to ensure the uniqueness and order of each transaction ID. Add the transaction ID to the data change record. When node A synchronizes data to node B, node B sorts the change records according to the transaction ID, executes the change operation of TID001 first, and then executes the change operation of TID002, so that the order of data changes on node B is consistent with that on node A.

[0147] By introducing a global transaction mechanism and generating a unique transaction ID for each change operation, the order of data change operations can be effectively managed and synchronized. The use of transaction IDs not only helps identify each change operation, but also ensures the consistency of the order of change operations in a distributed system, thereby avoiding data problems caused by inconsistent order.

[0148] In order to minimize the impact of data synchronization on system performance, asynchronous replication is used for data synchronization. The synchronization operation is asynchronous, and the change operation is cached through the message queue, allowing the master copy to respond to the client immediately after the change is completed without waiting for the synchronization to complete. At the same time, an efficient batch synchronization mechanism is designed to merge multiple change operations for synchronization, reduce the synchronization frequency, and improve the system throughput.

[0149] Asynchronous replication means that data change operations will not be synchronized to the replica nodes immediately, but will be performed through asynchronous replication. The master node will respond to the client immediately after completing the change, without waiting for the synchronization of the replica node to complete. Message queue cache is to put data change operations into the message queue for caching. The message queue is responsible for processing data synchronization in the background and sending the change operation from the master node to the replica node. Batch synchronization mechanism is to design a batch synchronization mechanism to merge multiple change operations into a batch for synchronization. By reducing the synchronization frequency, that is, not synchronizing every change immediately, the throughput and efficiency of the system can be improved. This reduces the impact of synchronization operations on system performance and improves the response speed and processing capacity of the system.

[0150] Reference Figure 2 and Figure 3 , a data storage system for land and resources planning is provided for an embodiment of the present application, the system comprising:

[0151] The information acquisition module is used to obtain the specified land and resources storage data, perform sharding on the stored data, and obtain data sharding information;

[0152] The shard analysis module is used to analyze the data shard information in real time, obtain hot data and cold data, perform synchronous replication on the hot data, and perform asynchronous replication on the cold data;

[0153] A partitioning module is used to obtain memory status information, partition the memory status information to obtain hierarchical information, adjust the hierarchical information according to hot data, and store the hot data in a plurality of hierarchical caches;

[0154] The dynamic adjustment module is used to obtain data access traffic information in real time, dynamically adjust the data sharding information according to the data access traffic information, and obtain distribution update information for updating hot data and cold data.

[0155] Specifically, as one implementation method, the specific steps of the shard analysis module performing synchronous replication processing on hotspot data are as follows:

[0156] The replication factor information and the hotspot synchronization frequency information are obtained, the hotspot data is replicated several times according to the replication factor information, and the replication rate of the hotspot data is adjusted according to the hotspot synchronization frequency information.

[0157] In one implementation mode, the specific steps of the shard analysis module performing asynchronous replication processing on cold data are as follows:

[0158] Acquire replication factor information and cold synchronization frequency information, replicate cold data several times according to the replication factor information, and adjust the replication rate of cold data according to the cold synchronization frequency information.

[0159] One embodiment of the present invention further includes:

[0160] The status analysis module obtains the operation index information in real time, analyzes the operation index information, and obtains the operation status information;

[0161] When the operation status information is judged to be in an abnormal mode, a backup copy is constructed based on the hierarchical cache, and the hot data and cold data are transferred and stored in the backup copy.

[0162] One embodiment of the present invention further includes:

[0163] The replica adjustment module is used to dynamically adjust the number of backup copies based on data access traffic information and operating indicator information. When the operating status information is judged to be an abnormal mode, the number of backup copies of the current hot data is increased; when the operating status information is judged to be a normal mode, the number of backup copies of the current cold data is reduced.

[0164] One embodiment of the present invention further includes:

[0165] The clock signal acquisition module is used to obtain vector clock information and change information of several nodes, mark the change information with a timestamp according to the vector clock information, and obtain change record information for recording the sequence.

[0166] One embodiment of the present invention further includes:

[0167] The node monitoring module is used to obtain node monitoring status information. When the node monitoring status information indicates a node failure, the hot data and cold data of the backup copy are obtained, and the hot data and cold data of the backup copy are restored to the hierarchical cache corresponding to the node failure.

[0168] One embodiment of the present invention further includes:

[0169] The global transaction module generates a transaction ID for the change information in the same change record information, and obtains the unique transaction ID information corresponding to the change information.

[0170] One embodiment of the present invention further includes:

[0171] The status analysis module is also used to run indicator information including CPU usage information, memory occupancy information, disk information and network traffic information.

[0172] Reference Figure 4 An embodiment of the present application provides a data storage device for land and resources planning, including a memory and a processor, the memory storing a computer program, and the processor being configured to run the computer program to execute the data storage method for land and resources planning as described above.

[0173] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the data storage method for land and resources planning as described above when running.

[0174] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and product can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0175] In several embodiments provided in this application, it should be understood that the disclosed methods, systems, devices and program products may be implemented in other ways.

[0176] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0177] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data storage method for land resources planning, characterized in that: include: Slice the land and resources data to obtain data slicing information; Perform real-time analysis on data sharding information to obtain hot data and cold data, perform synchronous replication on hot data, and perform asynchronous replication on cold data; Acquire memory status information, divide the memory status information to obtain hierarchical information, adjust the hierarchical information according to hot data, and store the hot data in a plurality of hierarchical caches; The data access traffic information is obtained in real time, and the data sharding information is dynamically adjusted according to the data access traffic information to obtain the distribution update information for updating the hot data and the cold data.

2. The data storage method for land resources planning according to claim 1 is characterized in that: The specific steps of synchronously replicating hotspot data are as follows: The replication factor information and the hotspot synchronization frequency information are obtained, the hotspot data is replicated several times according to the replication factor information, and the replication rate of the hotspot data is adjusted according to the hotspot synchronization frequency information.

3. The data storage method for land resources planning according to claim 1 is characterized in that: The specific steps of performing asynchronous replication processing on cold data are as follows: Acquire replication factor information and cold synchronization frequency information, replicate cold data several times according to the replication factor information, and adjust the replication rate of cold data according to the cold synchronization frequency information.

4. The data storage method for land resources planning according to claim 1 is characterized in that: Also includes: Obtain operation index information in real time, analyze the operation index information, and obtain operation status information; When the operation status information is judged to be in an abnormal mode, a backup copy is constructed based on the hierarchical cache, and the hot data and cold data are transferred and stored in the backup copy.

5. The data storage method for land resources planning according to claim 4 is characterized in that: Also includes: The number of backup copies is dynamically adjusted based on data access traffic information and operating indicator information. When the operating status information is judged to be an abnormal mode, the number of backup copies of the current hot data is increased. When the operating status information is judged to be a normal mode, the number of backup copies of the current cold data is reduced.

6. The data storage method for land resources planning according to claim 4 is characterized in that: Also includes: Vector clock information and change information of several nodes are obtained, and the change information is timestamped according to the vector clock information to obtain change record information for recording the sequence.

7. The data storage method for land resources planning according to claim 4 is characterized in that: Also includes: Obtain node monitoring status information, and when the node monitoring status information indicates a node failure, obtain hot data and cold data of the backup copy, and restore the hot data and cold data of the backup copy to the hierarchical cache corresponding to the node failure.

8. A data storage system for land resources planning, characterized in that: include: The information acquisition module is used to obtain the specified land and resources storage data, perform sharding on the stored data, and obtain data sharding information; The shard analysis module is used to analyze the data shard information in real time, obtain hot data and cold data, perform synchronous replication on the hot data, and perform asynchronous replication on the cold data; A partitioning module is used to obtain memory status information, partition the memory status information to obtain hierarchical information, adjust the hierarchical information according to hot data, and store the hot data in a plurality of hierarchical caches; The dynamic adjustment module is used to obtain data access traffic information in real time, dynamically adjust the data sharding information according to the data access traffic information, and obtain distribution update information for updating hot data and cold data.

9. A data storage device for land resources planning, characterized in that: It comprises a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the data storage method for land and resources planning described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the data storage method for land and resources planning as described in any one of claims 1-7 when running.

Images