Fine-grained disaster recovery method and device for analytical distributed database, and storage medium

Abstract

This invention discloses a fine-grained disaster recovery method, apparatus, and storage medium for analytical distributed databases. The method includes: globally uniquely mapping the data storage units of each storage node in the primary cluster to the storage locations in the backup cluster based on a pre-established storage layer mapping relationship between the primary and backup clusters; adjusting the synchronization frequency of the primary cluster according to the data change rate, periodically generating synchronization identifiers to identify the global data commit progress, and sending these identifiers to the backup cluster to segment the incremental data synchronization range; and the backup cluster pulling incremental data storage units belonging to the current synchronization range from the primary cluster based on the synchronization identifiers and data synchronization progress. Through block-level incremental identification and direct recovery, adaptive synchronization frequency adjustment, and storage layer mapping between heterogeneous clusters, the method reduces the target recovery point RPO while minimizing the performance impact on the primary cluster and effectively improving the resource utilization of the backup cluster.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a fine-grained disaster recovery method, apparatus, and storage medium for analytical distributed databases. Background Technology

[0002] With the rapid development of digital technologies, the application scenarios for massive amounts of data are constantly increasing, posing a severe challenge to data security. To address uncontrollable risks such as natural disasters, data center hardware failures, and human error, cross-regional multi-cluster storage architecture has become a core strategy for improving data resilience. This architecture constructs a redundant data storage system through geographically isolated multi-node deployment, ensuring that a single point of failure does not lead to global data loss. Even if a data center completely fails, nodes in other regions can quickly take over operations, guaranteeing zero data loss and business continuity. This type of disaster recovery mechanism does not rely on real-time synchronization but reduces the recovery time objective (RTO) through reasonable geographical distribution and backup strategies, replacing temporal redundancy with spatial redundancy to achieve high-availability storage.

[0003] Current disaster recovery solutions for analytical distributed databases primarily employ incremental data replication and recovery mechanisms based on SQL replay. This means the standby cluster synchronizes incremental data by replaying operation logs or SQL statements generated by the primary cluster. Additionally, some solutions support table-level disaster recovery configuration, allowing users to specify critical business tables for cross-cluster replication to reduce disaster recovery costs. In cross-regional deployment scenarios, primary and standby clusters are typically planned independently, often differing in the number of storage nodes and hardware resource configurations, resulting in a heterogeneous cluster environment.

[0004] However, the aforementioned existing technologies have the following drawbacks: First, the SQL replay-based recovery model is prone to performance bottlenecks when the incremental data scale surges. The overhead of log parsing, network transmission, and SQL replay is significant, making it difficult to reduce the recovery point target RPO and affecting the continuity of real-time analysis services. Second, when performing disaster recovery at the table level, the primary and backup clusters lack a dynamic resource adaptation mechanism due to their heterogeneous characteristics, such as varying numbers of nodes and different resource specifications. The backup cluster's storage resources cannot be used evenly, easily leading to situations where some nodes are overloaded while others are idle, resulting in low resource utilization and increased disaster recovery costs. Third, existing solutions lack fine-grained incremental identification and direct recovery capabilities at the data block level, making it difficult to achieve efficient and low-latency cross-cluster data synchronization while ensuring the performance of the primary cluster. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a fine-grained disaster recovery method, apparatus, and storage medium for analytical distributed databases. By using incremental identification and direct recovery at the data block level, adaptive synchronization frequency adjustment, and storage layer mapping between heterogeneous clusters, it reduces the target recovery point objective (RPO) while minimizing the performance impact on the primary cluster and effectively improving the resource utilization of the backup cluster.

[0006] In a first aspect, embodiments of the present invention provide a fine-grained disaster recovery method for analytical distributed databases. The method includes: based on a pre-established storage layer mapping relationship between a primary cluster and a backup cluster, globally uniquely mapping the data storage units of each storage node in the primary cluster to the storage locations in the backup cluster; the primary cluster adjusting the synchronization frequency according to the data change rate, periodically generating a synchronization identifier to identify the global data commit progress, and sending the synchronization identifier to the backup cluster to segment the incremental data synchronization range; the backup cluster pulling incremental data storage units belonging to the current synchronization range from the primary cluster based on the synchronization identifier and the data synchronization progress; wherein, the incremental data storage unit is obtained by the primary cluster by scanning the index file of the data storage units in its own cluster's storage nodes to locate the newly added data storage unit, and identifying it based on the matching result of the data version number recorded in the newly added data storage unit and the data version range corresponding to the synchronization identifier; the primary cluster attaching the corresponding globally unique identifier to the identified incremental data storage unit and directly sending it to the corresponding target storage node in the backup cluster for recovery according to the storage layer mapping relationship.

[0007] In one possible implementation, the storage layer mapping relationship establishes a bidirectional mapping rule based on the number of virtual storage nodes, data partition number, and data block number of the primary cluster and the backup cluster, so that the globally unique identifier of each data storage unit in the primary cluster corresponds one-to-one with the target storage location in the backup cluster.

[0008] In one possible implementation, the generation cycle of the synchronization identifier is on the order of seconds. The main cluster adaptively reduces or increases the generation frequency of the synchronization identifier based on the incremental data volume in the current cycle or the preset recovery point target threshold.

[0009] In one possible implementation, the index file is the Data Block Number Index (DPN) for appended data; the master cluster continues to scan the Data Block Number Index (DPN) from the last scan position to locate the newly added data storage unit.

[0010] In one possible implementation, the data version number is recorded in the dictionary DICT corresponding to the data storage unit; Matching results include: The master cluster compares the data version number recorded in the dictionary DICT with the data version range corresponding to the synchronization identifier. If the data version number falls within the data version range, it determines that the corresponding data storage unit belongs to the incremental data storage unit within the current synchronization range.

[0011] In one possible implementation, the backup cluster pulls incremental data storage units belonging to the current synchronization range from the primary cluster based on the synchronization identifier and data synchronization progress, including: The master node of the standby cluster generates a distributed data synchronization plan for the specified data object based on the synchronization identifier and data synchronization progress, and distributes the distributed data synchronization plan to each storage node of the standby cluster. Each storage node in the backup cluster pulls incremental data storage units from the corresponding storage node in the primary cluster in parallel according to the distributed data synchronization plan.

[0012] In one possible implementation, the primary cluster and the backup cluster are heterogeneous clusters, with different numbers of storage nodes or different resource specifications; or, the backup cluster includes a local backup cluster and a remote backup cluster, with storage layer mapping relationships established between the primary cluster and the local backup cluster, and between the primary cluster and the remote backup cluster, respectively.

[0013] Secondly, embodiments of the present invention provide a fine-grained disaster recovery device for analytical distributed databases, the device comprising: a mapping module, a generation module, and a retrieval module, wherein: The mapping module is used to globally and uniquely map the data storage units of each storage node in the primary cluster to the storage locations in the backup cluster based on the storage layer mapping relationship pre-established between the primary cluster and the backup cluster. The generation module is used by the primary cluster to adjust the synchronization frequency according to the data change rate, periodically generate synchronization identifiers to identify the global data submission progress, and send the synchronization identifiers to the backup cluster to divide the incremental data synchronization range. The pull module is used by the standby cluster to pull incremental data storage units that belong to the current synchronization range from the primary cluster based on the synchronization identifier and data synchronization progress. The incremental data storage unit is obtained by the primary cluster by scanning the index file of the data storage unit in the storage node of its own cluster to locate the new data storage unit, and identifying it according to the matching result of the data version number recorded in the new data storage unit and the data version range corresponding to the synchronization identifier. After the primary cluster attaches the corresponding globally unique identifier to the identified incremental data storage unit, it directly sends it to the corresponding target storage node in the standby cluster for recovery according to the storage layer mapping relationship.

[0014] Thirdly, embodiments of the present invention provide a computer storage medium storing multiple instructions adapted for loading by a processor and executing the steps of the above-described method.

[0015] Fourthly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being adapted to be loaded by the processor and to execute the steps of the above-described method.

[0016] The beneficial effects of the technical solutions provided by some embodiments of the present invention include at least the following: establishing a globally unique mapping relationship between data storage units in primary and backup clusters to achieve storage location adaptation in heterogeneous resource environments; adaptively adjusting the synchronization identifier generation frequency to reduce the performance impact of synchronization on the primary cluster while meeting the recovery point objective (RPO); and accurately identifying and directly sending data block-level incremental data storage units by scanning index files and combining this with data version number range matching. The present invention can effectively reduce RPO, ensure business continuity, improve the resource utilization of the backup cluster, and reduce the synchronization load on the primary cluster. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an exemplary system architecture diagram of a fine-grained disaster recovery system for an analytical distributed database provided in an embodiment of the present invention; Figure 2 A flowchart illustrating the fine-grained disaster recovery method for analytical distributed databases provided in this embodiment of the invention; Figure 3 A structural block diagram of a fine-grained disaster recovery device for an analytical distributed database provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the features and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present invention.

[0021] In the description of this invention, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances. Furthermore, in the description of this invention, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0022] As mentioned earlier, traditional disaster recovery solutions for analytical distributed databases often employ SQL replay-based data recovery models. These models are prone to performance bottlenecks when incremental data volumes surge, making it difficult to reduce the recovery point objective (RPO) and impacting the continuity of real-time analytics services. Furthermore, when performing disaster recovery at the table level, primary and backup clusters often exhibit heterogeneity, with varying numbers of nodes and different resource specifications. Existing technologies lack effective dynamic resource adaptation mechanisms, resulting in low utilization of backup cluster storage resources and high disaster recovery costs. In addition, the lack of fine-grained incremental identification and direct recovery capabilities at the data block level further restricts the balance between disaster recovery efficiency and primary cluster performance.

[0023] In view of this, the present invention provides a fine-grained disaster recovery method, apparatus, and storage medium for analytical distributed databases. By establishing a globally unique mapping relationship between data storage units in the primary and backup clusters, storage location adaptation in heterogeneous resource environments is achieved; by adaptively adjusting the synchronization identifier generation frequency, the performance impact of synchronization on the primary cluster is reduced while meeting the recovery point objective (RPO); by scanning the index file to locate newly added data storage units and combining this with data version number range matching, accurate incremental identification and direct transmission recovery at the data block level are achieved. The present invention can effectively reduce RPO, ensure business continuity, improve the resource utilization of the backup cluster, and reduce the synchronization load on the primary cluster, while simultaneously improving the RPO.

[0024] Please see Figure 1 , Figure 1 This is an exemplary system architecture diagram of a fine-grained disaster recovery method for analytical distributed databases provided in an embodiment of the present invention.

[0025] like Figure 1As shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 serves as the medium for providing a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired or wireless communication links, such as wired communication links including fiber optic cables, twisted-pair cables, or coaxial cables, and wireless communication links including Bluetooth communication links, Wireless-Fidelity (Wi-Fi) communication links, or microwave communication links, etc.

[0026] Terminal 101 can interact with server 103 via network 102 to receive messages from or send messages to server 103. Alternatively, terminal 101 can interact with server 103 via network 102 to receive messages or data sent to server 103 by other users. Terminal 101 can be hardware or software. When terminal 101 is hardware, it can be various electronic devices, including but not limited to smartwatches, smartphones, tablets, laptops, and desktop computers. When terminal 101 is software, it can be installed in the aforementioned electronic devices and can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module; no specific limitation is made here.

[0027] In this embodiment of the invention, terminal 101 can perform a globally unique mapping between the data storage units of each storage node in the primary cluster and the storage locations in the backup cluster based on the storage layer mapping relationship pre-established between the primary cluster and the backup cluster. The primary cluster adjusts the synchronization frequency according to the data change rate, periodically generates a synchronization identifier to identify the global data submission progress, and sends the synchronization identifier to the backup cluster to divide the incremental data synchronization range. Based on the synchronization identifier and the data synchronization progress, the backup cluster pulls incremental data storage units belonging to the current synchronization range from the primary cluster. The incremental data storage unit is obtained by the primary cluster by scanning the index file of the data storage units in the storage nodes of its own cluster to locate the new data storage unit, and by identifying it according to the matching result of the data version number recorded in the new data storage unit and the data version range corresponding to the synchronization identifier. After attaching the corresponding globally unique identifier to the identified incremental data storage unit, the primary cluster directly sends it to the corresponding target storage node in the backup cluster for recovery according to the storage layer mapping relationship.

[0028] Server 103 can be a business server providing various services. It should be noted that server 103 can be either hardware or software. When server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 103 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module; no specific limitations are made here.

[0029] Alternatively, the system architecture may not include server 103. In other words, server 103 may be an optional device in the embodiments of this specification. That is, the method provided in the embodiments of this specification can be applied to a system structure that only includes terminal 101. The embodiments of this invention do not limit this.

[0030] It should be understood that Figure 1 The number of terminals, networks, and servers shown is only illustrative; the number can be any number of terminals, networks, and servers depending on the implementation requirements.

[0031] Please see Figure 2 , Figure 2 This is a flowchart illustrating the fine-grained disaster recovery method for analytical distributed databases provided in this embodiment of the invention. The execution entity in this embodiment can be an electronic device executing the fine-grained disaster recovery for analytical distributed databases, a processor within the electronic device executing the fine-grained disaster recovery method for analytical distributed databases, or a fine-grained disaster recovery service for analytical distributed databases within the electronic device executing the fine-grained disaster recovery method for analytical distributed databases. For ease of description, the following uses a processor within an electronic device as an example to describe the specific execution process of the fine-grained disaster recovery method for analytical distributed databases.

[0032] like Figure 2 As shown, fine-grained disaster recovery methods for analytical distributed databases can include at least the following: S201. Based on the pre-established storage layer mapping relationship between the primary cluster and the backup cluster, a globally unique mapping is performed between the data storage units of each storage node in the primary cluster and the storage locations in the backup cluster.

[0033] Specifically, in cross-regional disaster recovery scenarios, the primary cluster and the backup cluster are often planned and deployed independently. They typically differ in the number of storage nodes and hardware resource configurations, forming a heterogeneous cluster environment. If data is directly copied to the backup cluster according to the primary cluster's storage structure, it can easily lead to some storage nodes in the backup cluster being overloaded while others remain idle, resulting in low resource utilization. To solve this problem, this invention pre-establishes a storage layer mapping relationship between the primary and backup clusters, uniquely mapping each data storage unit in the primary cluster to a target storage location in the backup cluster. Specifically, the storage resources of the primary and backup clusters are first virtualized. For the primary cluster, assuming it contains P physical storage nodes, and each physical storage node is configured to contain Q virtual storage slices, the total number of virtual storage nodes in the primary cluster is N = P × Q. Similarly, for the backup cluster, assuming it contains R physical storage nodes, and each physical storage node is also configured to contain Q virtual storage slices (the number of slices is kept consistent here to simplify the mapping logic, but different numbers of slices can be configured in actual applications and handled uniformly through subsequent mapping algorithms), the total number of virtual storage nodes in the backup cluster is M = R × Q.

[0034] Furthermore, the real storage nodes of the primary cluster are virtualized according to a preset number of storage slices, resulting in N virtual storage nodes; similarly, the real storage nodes of the backup cluster are virtualized according to a preset number of storage slices, resulting in M ​​virtual storage nodes. A globally unique identifier, denoted as DPID, is assigned to each data storage unit. DPID consists of three parts: virtual node number, data partition number, and data block number, and can be represented as (sliceId, partNo, DPNO). Here, sliceId is the virtual node number, determined by the real node number and the slice number within the node; partNo is the data partition number, which remains consistent between the primary and backup clusters; and DPNO is the data block number within the partition. For simplicity, this embodiment abbreviates DPID as (x, y), where x is the real node number, y is the data block number DPNO, and the data partition number partNo remains consistent between the primary and backup clusters and is implicit in the mapping rules. Let the globally unique identifier (DPID) of any data storage unit in the primary cluster be (x, y), where x is the actual node number and y is the data block number (DPNO). Let the globally unique identifier (DPID) of this DPID in the backup cluster be (x', y'), where x' is the target virtual storage node number and y' is the data block number within the target virtual storage node. The primary cluster has N virtual storage nodes, and the backup cluster has M virtual storage nodes. Then the mapping from (x, y) to (x', y') is:

[0035] The inverse mapping from (x', y') to (x, y) is:

[0036] The above mapping algorithm can ensure that even when the number of storage nodes in the primary and backup clusters is inconsistent and cannot be divided evenly, it can still maximize the resource utilization of the backup cluster under fine-grained disaster recovery and avoid data skew.

[0037] Optionally, for a two-site, three-center disaster recovery scenario, the backup cluster can include a local backup cluster and a remote backup cluster. Storage layer mapping relationships are established between the primary cluster and the local backup cluster, and between the primary cluster and the remote backup cluster, respectively. This allows primary cluster data to be simultaneously disaster-recovered to both local and remote target clusters. Furthermore, the two target clusters can independently configure mapping parameters according to their own resource specifications to meet different disaster recovery granularity requirements. Through the above settings, this embodiment lays the storage location foundation for the accurate sending and recovery of subsequent incremental data, ensuring load balancing across storage nodes in the backup cluster in a heterogeneous resource environment, effectively improving resource utilization, and reducing disaster recovery costs.

[0038] S202. The primary cluster adjusts the synchronization frequency according to the data change rate, periodically generates a synchronization identifier to identify the global data submission progress, and sends the synchronization identifier to the backup cluster to divide the incremental data synchronization range.

[0039] Specifically, in analytical databases, incremental data is continuously generated. If the synchronization frequency is too high, it will put significant performance pressure on the main cluster; if the frequency is too low, it will lead to an increase in the recovery point objective (RPO) and an increased risk of data loss. To address this issue, this embodiment uses the main cluster's master control node to monitor the data change rate in real time and dynamically adjust the generation frequency of synchronization flags based on the monitoring results, thereby achieving adaptive control of the synchronization load.

[0040] In one possible implementation, the generation cycle of synchronization identifiers is on the order of seconds. The primary cluster adaptively decreases or increases the generation frequency of synchronization identifiers based on the incremental data volume in the current cycle or a preset recovery point target threshold. Specifically, the primary cluster master node records the incremental data volume generated in each synchronization cycle. If the incremental data volume is lower than a preset low threshold, the generation interval of the next synchronization identifier is appropriately extended, for example, from 1 second to 2 seconds; if the incremental data volume is higher than a preset high threshold, the generation interval is shortened, for example, from 1 second to 500 milliseconds; if a fixed RPO target threshold is set, the generation interval of synchronization identifiers is ensured not to exceed the time window corresponding to that threshold. The synchronization identifier is a globally monotonically increasing transaction identifier (GID) used to mark the data version of all committed transactions before a certain point in time. The GID corresponds one-to-one with the primary cluster's global data version number (SCN). The SCN records the version information of each batch of committed data, while the GID is used to identify the global progress of the data commit at the cluster level. The primary cluster synchronizes the generated GID records to the backup cluster master node, allowing the backup cluster to know the data commit progress of the primary cluster. With the above settings, this embodiment minimizes the synchronization frequency and reduces interference with the query and analysis performance of the main cluster while meeting the business RPO requirements, thus achieving a dynamic balance between disaster recovery efficiency and main cluster load.

[0041] S203. The standby cluster pulls incremental data storage units that belong to the current synchronization range from the primary cluster based on the synchronization identifier and data synchronization progress.

[0042] Specifically, the standby cluster needs to acquire incremental data generated by the primary cluster within a specific GID range. However, scanning the entire dataset for each synchronization would incur huge overhead and fail to meet low RPO requirements. To address this issue, this embodiment employs a block-oriented incremental identification mechanism. The primary cluster storage nodes quickly locate newly added data storage units based on an index file and perform precise filtering using the data version number. In one possible implementation, the index file is the appended data block number index (DPN). The primary cluster continues scanning the DPN from the last scan position to locate newly added data storage units. Specifically, each storage node in the primary cluster maintains an appended DPN file, recording coarse-grained index information for each data storage unit (DP). The DPN file is only updated and not modified. During each synchronization, the newly added DPN records are read from the position where the last scan ended, thus quickly identifying which DPs have been added without traversing all the data. In another possible implementation, the data version number is recorded in the dictionary DICT corresponding to the data storage unit. The matching result is as follows: the main cluster compares the data version number recorded in the dictionary DICT with the data version range corresponding to the synchronization identifier. If the data version number falls within the data version range, the corresponding data storage unit is determined to be an incremental data storage unit within the current synchronization range. Specifically, each DP corresponds to a DICT dictionary entry, and the DICT records the latest data version number (SCN) of the data contained in the DP. When a new DP is located through the DPN, its SCN in the DICT is read, and it is determined whether the SCN is within the range of (previous synchronization GID, current synchronization GID). If it is, the DP is marked as incremental data to be synchronized. Through the two-level filtering of DPN fast location and DICT precise filtering, the main cluster can efficiently and accurately identify incremental data blocks within the current GID range. After the identified incremental data storage unit is appended with its globally unique identifier (DPID), it is directly sent to the corresponding target storage node in the backup cluster according to the storage layer mapping relationship established in S201. In one possible implementation, the standby cluster retrieves incremental data storage units belonging to the current synchronization range from the primary cluster based on synchronization identifiers and data synchronization progress. This includes: the primary control node of the standby cluster generating a distributed data synchronization plan for a specified data object based on the synchronization identifier and data synchronization progress, and distributing the distributed data synchronization plan to each storage node of the standby cluster; each storage node of the standby cluster retrieving incremental data storage units from the corresponding storage node of the primary cluster in parallel according to the distributed data synchronization plan. Specifically, the primary control node of the standby cluster generates a distributed synchronization plan based on the currently synchronized GID progress and the range of GIDs to be synchronized this time, combined with the distribution of data objects, specifying which DPID data each storage node of the standby cluster needs to retrieve from which storage nodes of the primary cluster; after receiving the plan, each storage node of the standby cluster concurrently initiates data retrieval requests to the corresponding node of the primary cluster, achieving parallel and efficient recovery.After receiving the DP data and the corresponding DPN data, the target storage node of the backup cluster writes the data to the corresponding storage location according to the DPID and the local storage structure, thus completing this incremental recovery.

[0043] This invention provides a fine-grained disaster recovery method for analytical distributed databases. It achieves accurate identification and direct recovery of incremental data at the data block level, effectively reducing the target recovery point objective (RPO) and improving disaster recovery efficiency. By adaptively adjusting the synchronization identifier generation frequency, it reduces the performance impact on the primary cluster while meeting RPO requirements. By establishing a bidirectional storage layer mapping relationship between the primary and backup clusters based on virtual node mapping, it solves the problem of heterogeneous cluster resource adaptation, resulting in balanced load and significantly improved resource utilization across storage nodes in the backup cluster. Furthermore, a two-level filtering mechanism—rapid location using DPN index and precise filtering using DICT version numbers—along with distributed parallel pull recovery in the backup cluster, further optimizes the accuracy and speed of incremental data identification. In addition, this solution supports multi-target cluster heterogeneous adaptation in complex disaster recovery scenarios such as two-site, three-center setups, demonstrating broad applicability. This invention effectively solves the technical problems of low recovery efficiency, high primary cluster load, uneven backup cluster resource utilization, and insufficient incremental identification accuracy in existing technologies, achieving outstanding advantages of low RPO, low impact, and high resource utilization.

[0044] Please see Figure 3 , Figure 3 This is a structural block diagram of a fine-grained disaster recovery device for an analytical distributed database provided in an embodiment of the present invention. Figure 3 As shown: The fine-grained disaster recovery device 300 for analytical distributed databases includes: a mapping module 310, a generation module 320, and a pull module 330, wherein: The mapping module 310 is used to globally and uniquely map the data storage units of each storage node in the primary cluster to the storage locations in the backup cluster based on the storage layer mapping relationship pre-established between the primary cluster and the backup cluster. The generation module 320 is used by the primary cluster to adjust the synchronization frequency according to the data change rate, periodically generate a synchronization identifier to identify the global data submission progress, and send the synchronization identifier to the backup cluster to divide the incremental data synchronization range. The pull module 330 is used by the standby cluster to pull incremental data storage units belonging to the current synchronization range from the primary cluster based on the synchronization identifier and data synchronization progress. The incremental data storage unit is obtained by the primary cluster by scanning the index file of the data storage unit in the storage node of its own cluster to locate the new data storage unit, and identifying it according to the matching result of the data version number recorded in the new data storage unit and the data version range corresponding to the synchronization identifier. After the primary cluster attaches the corresponding globally unique identifier to the identified incremental data storage unit, it sends it directly to the corresponding target storage node in the standby cluster for recovery according to the storage layer mapping relationship.

[0045] In one possible embodiment, the storage layer mapping relationship establishes a bidirectional mapping rule based on the number of virtual storage nodes, data partition number, and data block number of the primary cluster and the backup cluster, so that the globally unique identifier of each data storage unit in the primary cluster corresponds one-to-one with the target storage location in the backup cluster.

[0046] In one possible embodiment, the generation cycle of the synchronization identifier is on the order of seconds. The main cluster adaptively reduces or increases the generation frequency of the synchronization identifier based on the incremental data volume in the current cycle or the preset recovery point target threshold.

[0047] In one possible implementation, the index file is the Data Block Number Index (DPN) for appended data; the master cluster continues to scan the Data Block Number Index (DPN) from the last scan position to locate the newly added data storage unit.

[0048] In one possible embodiment, the data version number is recorded in the dictionary DICT corresponding to the data storage unit; Matching results include: The master cluster compares the data version number recorded in the dictionary DICT with the data version range corresponding to the synchronization identifier. If the data version number falls within the data version range, it determines that the corresponding data storage unit belongs to the incremental data storage unit within the current synchronization range.

[0049] In one possible embodiment, the pull module 330 includes: The distribution unit is used by the master control node of the standby cluster to generate a distributed data synchronization plan for a specified data object based on the synchronization identifier and data synchronization progress, and then distribute the distributed data synchronization plan to each storage node of the standby cluster. The pull unit is used by each storage node of the standby cluster to pull incremental data storage units from the corresponding storage nodes of the primary cluster in parallel according to the distributed data synchronization plan.

[0050] In one possible embodiment, the primary cluster and the backup cluster are heterogeneous clusters, with different numbers of storage nodes or different resource specifications; or, the backup cluster includes a local backup cluster and a remote backup cluster, with storage layer mapping relationships established between the primary cluster and the local backup cluster, and between the primary cluster and the remote backup cluster, respectively.

[0051] It should be noted that the fine-grained disaster recovery device for analytical distributed databases provided in the above embodiments is only illustrated by the division of the above functional modules when executing the fine-grained disaster recovery method for analytical distributed databases. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the fine-grained disaster recovery device for analytical distributed databases and the fine-grained disaster recovery method embodiments for analytical distributed databases belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0052] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0053] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 4 As shown, the electronic device 400 may include: at least one processor 401, at least one network interface 404, user interface 403, memory 405, and at least one communication bus 402.

[0054] The communication bus 402 is used to enable communication between these components.

[0055] The user interface 403 may include a display screen, and the optional user interface 403 may include a standard wired interface or a wireless interface.

[0056] The network interface 404 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0057] The processor 401 may include one or more processing cores. The processor 401 connects to various parts within the electronic device 400 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 405, and by calling data stored in the memory 405. Optionally, the processor 401 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 401 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 401.

[0058] The memory 405 may include random access memory (RAM) or read-only memory. Optionally, the memory 405 may include a non-transitory computer-readable storage medium. The memory 405 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 405 may also be at least one storage device located remotely from the aforementioned processor 401. Figure 4 As shown, the memory 405, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a fine-grained disaster recovery application for an analytical distributed database.

[0059] exist Figure 4In the illustrated electronic device 400, the user interface 403 is mainly used to provide an input interface for the user and to obtain the user's input data; while the processor 401 can be used to call the fine-grained disaster recovery application of the analytical distributed database stored in the memory 405, and specifically perform the following operations: Based on the pre-established storage layer mapping relationship between the primary and backup clusters, the data storage units of each storage node in the primary cluster are globally uniquely mapped to the storage locations in the backup cluster. The primary cluster adjusts the synchronization frequency according to the data change rate, periodically generates a synchronization identifier to identify the global data commit progress, and sends the synchronization identifier to the backup cluster to divide the incremental data synchronization range. Based on the synchronization identifier and data synchronization progress, the backup cluster pulls incremental data storage units belonging to the current synchronization range from the primary cluster. The incremental data storage unit is identified by the primary cluster by scanning the index file of the data storage units in the storage nodes of its own cluster to locate the new data storage unit, and by matching the data version number recorded in the new data storage unit with the data version range corresponding to the synchronization identifier. After attaching the corresponding globally unique identifier to the identified incremental data storage unit, the primary cluster directly sends it to the corresponding target storage node in the backup cluster for recovery according to the storage layer mapping relationship.

[0060] In some possible embodiments, the storage layer mapping relationship establishes a bidirectional mapping rule based on the number of virtual storage nodes, data partition number, and data block number of the primary cluster and the backup cluster, so that the globally unique identifier of each data storage unit in the primary cluster corresponds one-to-one with the target storage location in the backup cluster.

[0061] In some possible embodiments, the generation cycle of the synchronization identifier is on the order of seconds. The main cluster adaptively reduces or increases the generation frequency of the synchronization identifier based on the incremental data volume in the current cycle or the preset recovery point target threshold.

[0062] In some possible implementations, the index file is the Data Block Number Index (DPN) for appended data; the master cluster continues scanning the Data Block Number Index (DPN) from the last scan position to locate the newly added data storage unit.

[0063] In some possible embodiments, the data version number is recorded in the dictionary DICT corresponding to the data storage unit; the matching result includes: the master cluster compares the data version number recorded in the dictionary DICT with the data version range corresponding to the synchronization identifier. If the data version number falls within the data version range, it is determined that the corresponding data storage unit belongs to the incremental data storage unit within the current synchronization range.

[0064] In some possible embodiments, the processor 401 executes the standby cluster to pull incremental data storage units belonging to the current synchronization range from the primary cluster based on the synchronization identifier and data synchronization progress, specifically for the following purposes: The master node of the standby cluster generates a distributed data synchronization plan for the specified data object based on the synchronization identifier and data synchronization progress, and distributes the distributed data synchronization plan to each storage node of the standby cluster. Each storage node in the backup cluster pulls incremental data storage units from the corresponding storage node in the primary cluster in parallel according to the distributed data synchronization plan.

[0065] In some possible embodiments, the primary cluster and the backup cluster are heterogeneous clusters, with different numbers of storage nodes or different resource specifications; or, the backup cluster includes a local backup cluster and a remote backup cluster, with storage layer mapping relationships established between the primary cluster and the local backup cluster, and between the primary cluster and the remote backup cluster, respectively.

[0066] This invention also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the above-described instructions. Figure 2 One or more steps in the illustrated embodiment. If the constituent modules of the fine-grained disaster recovery device for the above-described analytical distributed database are implemented as software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0067] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid state disks (SSDs)).

[0068] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation schemes can be combined arbitrarily.

[0069] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. It will be apparent to those skilled in the art that various modifications can be made to these embodiments, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A fine-grained disaster recovery method for analytical distributed databases, characterized in that, The method includes: Based on the pre-established storage layer mapping relationship between the primary cluster and the backup cluster, the data storage units of each storage node in the primary cluster are globally and uniquely mapped to the storage locations in the backup cluster. The primary cluster adjusts the synchronization frequency according to the data change rate, periodically generates a synchronization identifier to identify the global data submission progress, and sends the synchronization identifier to the backup cluster to divide the incremental data synchronization range. The backup cluster retrieves incremental data storage units belonging to the current synchronization range from the primary cluster based on the synchronization identifier and data synchronization progress. The incremental data storage units are obtained by the primary cluster by scanning the index files of data storage units in its own storage nodes to locate new data storage units and identifying them based on the matching result of the data version number recorded in the new data storage unit and the data version range corresponding to the synchronization identifier. The primary cluster attaches a corresponding globally unique identifier to the identified incremental data storage units and sends them directly to the corresponding target storage node in the backup cluster for recovery according to the storage layer mapping relationship.

2. The fine-grained disaster recovery method for analytical distributed databases according to claim 1, characterized in that, The storage layer mapping relationship establishes a bidirectional mapping rule based on the number of virtual storage nodes, data partition number, and data block number of the primary cluster and the backup cluster, so that the globally unique identifier of each data storage unit in the primary cluster corresponds one-to-one with the target storage location in the backup cluster.

3. The fine-grained disaster recovery method for analytical distributed databases according to claim 1, characterized in that, The generation cycle of the synchronization identifier is on the order of seconds. The main cluster adaptively reduces or increases the generation frequency of the synchronization identifier based on the incremental data volume in the current cycle or the preset recovery point target threshold.

4. The fine-grained disaster recovery method for analytical distributed databases according to claim 1, characterized in that, The index file is the Data Block Number Index (DPN) for appending data; the main cluster continues to scan the Data Block Number Index (DPN) from the last scan position to locate the newly added data storage unit.

5. The fine-grained disaster recovery method for analytical distributed databases according to claim 1, characterized in that, The data version number is recorded in the dictionary DICT corresponding to the data storage unit; The matching results include: The master cluster compares the data version number recorded in the dictionary DICT with the data version range corresponding to the synchronization identifier. If the data version number falls within the data version range, it determines that the corresponding data storage unit belongs to the incremental data storage unit within the current synchronization range.

6. The fine-grained disaster recovery method for analytical distributed databases according to claim 1, characterized in that, The backup cluster, based on the synchronization identifier and data synchronization progress, pulls incremental data storage units belonging to the current synchronization range from the primary cluster, including: The master control node of the backup cluster generates a distributed data synchronization plan for a specified data object based on the synchronization identifier and data synchronization progress, and distributes the distributed data synchronization plan to each storage node of the backup cluster. Each storage node in the backup cluster pulls incremental data storage units from the corresponding storage node of the primary cluster in parallel according to the distributed data synchronization plan.

7. The fine-grained disaster recovery method for analytical distributed databases according to claim 1, characterized in that, The primary cluster and the backup cluster are heterogeneous clusters, with different numbers of storage nodes or different resource specifications. Alternatively, the backup cluster may include a local backup cluster and a remote backup cluster, and the storage layer mapping relationship may be established between the primary cluster and the local backup cluster, and between the primary cluster and the remote backup cluster, respectively.

8. A fine-grained disaster recovery device for an analytical distributed database, characterized in that, The device includes: The mapping module is used to globally and uniquely map the data storage units of each storage node in the primary cluster to the storage locations in the backup cluster based on the storage layer mapping relationship pre-established between the primary cluster and the backup cluster. The generation module is used by the primary cluster to adjust the synchronization frequency according to the data change rate, periodically generate a synchronization identifier to identify the global data submission progress, and send the synchronization identifier to the backup cluster to divide the incremental data synchronization range. The pull module is used by the backup cluster to pull incremental data storage units belonging to the current synchronization range from the primary cluster based on the synchronization identifier and data synchronization progress. The incremental data storage units are obtained by the primary cluster by scanning the index files of data storage units in the storage nodes of the cluster to locate the new data storage units and identifying them according to the matching result of the data version number recorded in the new data storage units and the data version range corresponding to the synchronization identifier. After attaching the corresponding globally unique identifier to the identified incremental data storage units, the primary cluster directly sends them to the corresponding target storage node in the backup cluster for recovery according to the storage layer mapping relationship.

9. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the steps of the method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method as described in any one of claims 1 to 7.

Images