Method and system for realizing data and task disaster recovery across multiple hive clusters

Abstract

The application belongs to the field of big data application, and discloses a method and system for realizing data and task disaster recovery across multiple Hive clusters, comprising the following steps: obtaining cross-domain mutual trust of kerberos authentication of Hadoop cluster A and Hadoop cluster B; obtaining cluster data of Hadoop cluster A and cluster data of Hadoop cluster B, and copying and synchronizing each other; when data loss or damage occurs between the two, scheduling and repairing between the two, compared with a single Hive cluster, achieving customer-unaware data and storage and computing task disaster recovery, guaranteeing business continuity, avoiding the destruction of regional disasters on enterprise data and business, greatly improving the availability and reliability of the cluster, so as to reduce or avoid the impact and loss caused by business interruption and data loss.

Description

Technical Field

[0001] This invention relates to the field of big data applications, and more specifically, to a method and system for data and task disaster recovery across multiple Hive clusters. Background Technology

[0002] Currently, big data has four characteristics: Volume, Variety, Velocity, and Value, which are generally referred to as the 4Vs.

[0003] Hive plays a very important role in offline data / batch data storage and computing. It can be said that batch processing scenarios / data warehouse construction are basically inseparable from Hive, and Hive data warehouses often store massive amounts of data.

[0004] Furthermore, for IT companies, traditional single data centers are no longer sufficient to protect enterprise data security. When a single data center storage fails, it may lead to prolonged business interruption or even data loss. Local data redundancy protection or disaster recovery construction alone cannot prevent regional disasters from damaging enterprise data. Remote disaster recovery to protect data and ensure business continuity have become urgent issues for enterprises. Therefore, the needs for data disaster recovery / synchronization and the continuity of storage and computing tasks for Hive clusters across IDCs / regions / data centers are particularly urgent and important. Summary of the Invention

[0005] To overcome the aforementioned deficiencies in the prior art, this invention provides a method for data and task disaster recovery across multiple Hive clusters, comprising the following steps: Step 1: Obtain Kerberos authentication for Hadoop cluster A and Hadoop cluster B, and establish cross-domain mutual trust; Step 2: Obtain cluster data from Hadoop cluster A and Hadoop cluster B, and perform mutual replication and synchronization; Step 3: If cluster data from Hadoop cluster A is lost or corrupted, receive cluster data scheduling from Hadoop cluster B; if cluster data from Hadoop cluster B is lost or corrupted, receive cluster data scheduling from Hadoop cluster A; Step 4: If cluster data from Hadoop cluster A is lost or corrupted, receive cluster data from Hadoop cluster B to repair cluster data from Hadoop cluster A; if cluster data from Hadoop cluster B is lost or corrupted, receive cluster data from Hadoop cluster A to repair cluster data from Hadoop cluster B.

[0006] Preferably, the method for synchronizing data between Hadoop cluster A and Hadoop cluster B includes the following steps:

[0007] The server disks containing the metadata database (MySQL) of the S100, cross-IDC / city Hive cluster are configured as RAID1 / RAID10.

[0008] S101. Configure cross-domain mutual trust for Kerberos authentication between Hadoop cluster A and Hadoop cluster B, and synchronize tenant information.

[0009] S102. Obtain the cluster health status serverState;

[0010] S103. Use the canal tool to synchronize Hive metadata in the MySQL metadata database of two Hive clusters in real time.

[0011] S104. Verify the Hive metadata synchronization result. If the verification fails, generate an error alarm and return to S103. If the verification is normal, proceed to S105.

[0012] S105. Use the distcp script tool to synchronize HDFS data;

[0013] S106. HDFS synchronization result data consistency verification. If the verification is abnormal, an abnormal alarm is generated and returned to S105. If the verification is normal, proceed to S107.

[0014] S107. Data synchronization between the two Hive clusters is complete.

[0015] Preferably, the Hive metadata synchronization result verification method includes:

[0016] The table structure and metadata, database objects, table data statistics, partition information, and Hive configuration information of Hadoop cluster A and Hadoop cluster B are compared and their consistency is verified.

[0017] If they match, it is considered normal;

[0018] If there is a discrepancy, an error alert will be generated.

[0019] Preferably, the method for verifying the consistency of HDFS synchronization results is as follows: obtain the file lists and file contents of Hadoop cluster A and Hadoop cluster B respectively;

[0020] Compare the file lists of Hadoop cluster A and Hadoop cluster B to obtain the file lists on the source HDFS and target HDFS, including file paths, file sizes, and modification times. By comparing the file lists on the two clusters, the consistency of files between the source and target is ensured.

[0021] Compare the hash values ​​of the file contents in Hadoop cluster A and Hadoop cluster B. If the hash values ​​are the same, the file contents are consistent; otherwise, they are inconsistent.

[0022] Preferably, the method for monitoring failures in Hadoop cluster A and Hadoop cluster B includes the following steps:

[0023] Get the serverState values ​​of Hadoop cluster A and Hadoop cluster B respectively. Hadoop cluster A is marked as cluster1 and Hadoop cluster B is marked as cluster2. Cluster1 serves as the master data service and is normally the one that provides the service.

[0024] serverState=0 means that both cluster1 and cluster2 are providing normal service;

[0025] serverState=1 means that only cluster1 is faulty;

[0026] serverState=2 means that only cluster2 is faulty;

[0027] serverState=3 means that both cluster1 and cluster2 are down.

[0028] Preferably, when cluster1 fails and cluster2 is normal, the handling method is as follows:

[0029] When the value of serverState in cluster2 is equal to 0, the data disaster recovery direction is cluster1->cluster2. Hive metadata between Hadoop cluster A and Hadoop cluster B is synchronized in real time, including:

[0030] Period T triggers incremental data synchronization between database tables;

[0031] Triggered upon completion of the storage and computing scheduling task, when the incremental database table data is greater than M, HDFS incremental file synchronization is executed, where M is the value of network bandwidth S between cluster1 and cluster2 multiplied by 4 hours.

[0032] When the value of serverState in cluster2 is equal to 1, the data disaster recovery direction is cluster1->cluster2, and the Hive metadata of Hadoop cluster A and Hadoop cluster B are synchronized in real time.

[0033] Preferably, when cluster1 recovers from a fault and cluster2 remains normal, the data disaster recovery direction is cluster2->cluster1. The processing methods are divided into two types: executing based on the failure of the cluster1 metadata database and using Canal to perform Hive metadata synchronization between cluster1 and cluster2.

[0034] S200: If the cluster1 metadata database is normal, proceed directly to S201. If the cluster1 metadata database fails, obtain the timestamp of the latest data in the BinLog of the MySQL metadata database of cluster1 and cluster2 at the current moment, extract the historical changes of metadata of cluster2 during the failure of cluster1 according to the timestamp, and perform synchronization on cluster1.

[0035] S201. Using Hive's metadata / MySQL's binlog, parse out the incremental information of Hive database table data during the period from cluster1 failure to recovery;

[0036] S202. Use the distcp script to synchronize incremental information to cluster1;

[0037] S203, Data consistency check. If it fails, return to S202 loop. If it succeeds, proceed to S206.

[0038] S204. Using Canal to perform Hive metadata synchronization between cluster1 and cluster2 is divided into:

[0039] Period T triggers data synchronization for the newly added database table;

[0040] Triggered upon completion of the weekly storage and computation task, when the incremental database table data exceeds M, HDFS incremental file synchronization is executed;

[0041] S205, Data consistency check. If it fails, return to S204 and loop. If it succeeds, proceed to S206 to execute.

[0042] S206, Data consistency verification during the start and recovery times of cluster1 failure;

[0043] S207 If the verification passes, the cluster service can be switched from cluster2 to cluster1. If the verification fails, return to S205 or S203 in a loop.

[0044] Preferably, when cluster1 is always normal and cluster2 recovers from a fault, the data disaster recovery direction is cluster1->cluster2. The processing methods are divided into execution based on the cluster2 metadata database failure and using Canal to perform Hive metadata synchronization between cluster1 and cluster2.

[0045] S300: If the cluster1 metadata database is normal, proceed directly to S301. If the cluster2 metadata database fails, obtain the timestamp of the latest data in the BinLog of the MySQL metadata database of cluster1 and cluster2 at the current moment, extract the historical changes of metadata of cluster1 during the failure of cluster2 according to the timestamp, and perform synchronization on cluster2.

[0046] S301. Using Hive's metadata / MySQL's binlog, parse out the incremental information of Hive database table data during the period from cluster2 failure to recovery;

[0047] S302. Use the distcp script to synchronize incremental information to cluster2;

[0048] S303, Data consistency check. If it fails, return to S302 loop. If it succeeds, proceed to S306.

[0049] S304. Using Canal to perform Hive metadata synchronization between cluster1 and cluster2 is divided into:

[0050] Period T triggers data synchronization for the newly added database table;

[0051] Triggered upon completion of the weekly storage and computation task, when the incremental database table data exceeds M, HDFS incremental file synchronization is executed;

[0052] S305, Data consistency check. If it fails, return to S304 and loop. If it succeeds, proceed to S306 to execute.

[0053] S306, during the start and recovery times of cluster2 failure, data consistency is checked. If the check passes, the process ends; otherwise, it returns to the S305 or S303 loop.

[0054] A data and task disaster recovery system that spans multiple Hive clusters is used to implement the above-mentioned data and task disaster recovery method that spans multiple Hive clusters. It includes a cluster monitoring Listener module and a unified proxy module, and is deployed on a cloud with high availability and security, outside of cluster1 and cluster2.

[0055] The cluster monitoring Listener module sends test scripts at time intervals of T, traverses the metadata database, Hive database, and HDFS cluster of test cluster1 and cluster2, and obtains the health status of the two clusters and serverState, where T is an integer greater than or equal to 1.

[0056] If a failure is detected in cluster1 or cluster2, data disaster recovery will be performed, and the task scheduling cluster will be replaced by a unified proxy module.

[0057] Preferably, if the cluster monitoring Listener module detects that 1 / 3 of the DataNode nodes are faulty, it will switch between cluster1 and cluster2, and correspondingly switch between data disaster recovery and task scheduling.

[0058] In the event of a failure in cluster1 or cluster2, the cluster monitoring Listener module performs cluster health status detection according to the incrementing failure period T = T + K * detection result return time, where K is a constant between (0,1) and T ≤ 1h; if cluster1 or cluster2 recovers to normal, T is set to the initial value.

[0059] The technical effects and advantages of this invention, which describes a method and system for data and task disaster recovery across multiple Hive clusters, are as follows:

[0060] Compared to a single Hive cluster, it enables seamless disaster recovery of data and storage tasks for customers, ensuring business continuity and mitigating the damage to enterprise data and business caused by regional disasters. This greatly improves the availability and reliability of the cluster, reducing or avoiding the impact and losses that may result from business interruptions and data loss. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the overall disaster recovery of the off-site Hive cluster according to the present invention;

[0062] Figure 2 This is a flowchart illustrating the overall process of data synchronization between clusters in this invention.

[0063] Figure 3 This is the overall flowchart for ensuring the continuity of tasks such as storage and computation in this invention;

[0064] Figure 4 This is a schematic diagram of the cluster data disaster recovery and restoration mechanism of the present invention;

[0065] Figure 5 This is a schematic diagram of the main cluster fault recovery process of the present invention;

[0066] Figure 6This is a schematic diagram of the cluster fault recovery process of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] This invention provides, for example Figure 1-6 This embodiment illustrates a method for data and task disaster recovery across multiple Hive clusters. It uses data disaster recovery / synchronization across two geographically dispersed Hive clusters as an example, specifically Hadoop cluster A and Hadoop cluster B. The architectures of Hadoop cluster A and Hadoop cluster B are identical. (See reference...) Figure 1 As shown, taking Hadoop cluster A as an example, the Hive metadata database MySQL adopts a master-slave distributed architecture (one master and multiple slaves are possible). That is, the servers where Hadoop cluster A and Hadoop cluster B, mysql-master and mysql-slave are located are distributed in different racks within the same IDC, and the corresponding server disks are configured with RAID0 / RAID10 to ensure disaster recovery and high availability of Hive metadata within a single data center.

[0069] The HDFS cluster that Hive relies on adopts an HA architecture. A dedicated network line is used between Hadoop cluster A and Hadoop cluster B. The specific bandwidth depends on the actual needs. Kerberos authentication cross-domain mutual trust is configured between Hadoop cluster A and Hadoop cluster B.

[0070] To obtain data from Hadoop cluster A and Hadoop cluster B, and to replicate and synchronize the data between them, a data replication tool such as Apache Kafka, Apache NiFi, or a custom script is used to achieve data replication and synchronization between Hadoop cluster A and Hadoop cluster B, thus maintaining data consistency.

[0071] If the data in Hadoop cluster A is lost or corrupted, it will receive data from Hadoop cluster B for scheduling; if the data in Hadoop cluster B is lost or corrupted, it will receive data from Hadoop cluster A for scheduling. Task scheduling uses a distributed task scheduling system, such as Apache Airflow, to manage and schedule tasks, ensuring that tasks can be switched between Hadoop cluster A and Hadoop cluster B for disaster recovery.

[0072] If the data in Hadoop cluster A is lost or corrupted, it will receive data from Hadoop cluster B to repair the data in Hadoop cluster A.

[0073] See Figure 2 As shown, the method for synchronizing data between Hadoop cluster A and Hadoop cluster B includes the following steps:

[0074] The S100 Hive cluster's metadata database (MySQL) server disk is configured as RAID1 / RAID10. RAID1 provides data mirroring, writing data to multiple disks simultaneously to improve data redundancy and read performance. RAID10 mirrors and stripes multiple disks, providing even higher redundancy and better read / write performance.

[0075] S101. Configure Kerberos authentication cross-domain mutual trust between Hadoop cluster A and Hadoop cluster B, and synchronize tenant and other related information. That is, Hadoop cluster A and Hadoop cluster B authenticate and trust each other, so that cross-domain users can access and operate between Hadoop cluster A and Hadoop cluster B.

[0076] S102. Obtain the cluster health status serverState;

[0077] S103. Use the Canal tool to synchronize Hive metadata in the MySQL metadata database of two Hive clusters in real time. Canal is an open-source MySQL database binlog parsing tool used to parse MySQL binlog logs and synchronize data changes to other target databases in real time. By configuring the Canal tool, the Hive metadata changes in cluster A are synchronized to the Hive metadata database in cluster B in real time, ensuring that the metadata of the two clusters are consistent.

[0078] S104. Verify the Hive metadata synchronization result. If the verification fails, generate an error alarm and return to S103. If the verification is normal, proceed to S105.

[0079] S105. Use the distcp script tool to perform HDFS data synchronization. Use the distcp script to realize HDFS data synchronization between Hadoop cluster A and Hadoop cluster B. distcp is a tool provided by Hadoop for distributed data replication. It can replicate data between different Hadoop clusters and maintain data consistency. By configuring the corresponding parameters and paths, data in Hadoop cluster A can be copied to Hadoop cluster B to achieve data synchronization.

[0080] S106. HDFS synchronization result data consistency verification. If the verification is abnormal, an abnormal alarm is generated and returned to S105. If the verification is normal, proceed to S107.

[0081] S107. Data synchronization between the two Hive clusters is complete.

[0082] Furthermore, the Hive metadata synchronization result verification method includes: comparing the table structure and metadata, database objects, table data statistics, partition information, and Hive configuration information of Hadoop cluster A and Hadoop cluster B respectively; if they are consistent, it is considered normal; if they are inconsistent, an abnormal alarm is generated.

[0083] Furthermore, the HDFS synchronization result data consistency verification method includes: file list comparison: obtaining file lists from the source HDFS and target HDFS, including file paths, file sizes, modification times, and other information. By comparing the file lists on the two clusters, the consistency of files between the source and target is ensured; file content comparison: selecting some representative files or sample files, obtaining their contents from the source HDFS and target HDFS respectively, and comparing the contents. This can be achieved by calculating the hash value of the files (such as MD5 or SHA1). If the file content is consistent, the hash values ​​will be the same; otherwise, it indicates a problem with data synchronization. Specific comparison and verification also include directory structure comparison, obtaining the directory structure on the source HDFS and target HDFS, including directory names, subdirectories, and file hierarchy. By comparing the directory structures on the two clusters, it is ensured that the directory hierarchy and structure are consistent between the source and target. File permission and attribute comparison involves obtaining and comparing the permissions, owner, modification time, access time, and other attributes of files on the source and target HDFS to ensure that file permissions and attributes are consistent between the source and target. Logs and reports record errors, differences, and anomalies during the comparison process and generate corresponding logs and reports for further analysis and repair. Specific comparison items are configured by those skilled in the art according to actual needs.

[0084] See Figure 3As shown, the monitoring method for faults in Hadoop cluster A and Hadoop cluster B includes the following steps: obtaining the serverState values ​​of Hadoop cluster A and Hadoop cluster B respectively. Hadoop cluster A is marked as cluster1 and Hadoop cluster B is marked as cluster2. Cluster1 serves as the master data service and is normally the service provider.

[0085] serverState=0 means that both cluster1 and cluster2 are providing normal service;

[0086] serverState=1 means that only cluster1 is faulty;

[0087] serverState=2 means that only cluster2 is faulty;

[0088] serverState=3 means that both cluster1 and cluster2 are down;

[0089] By monitoring the value of serverState, it is possible to clearly determine whether Hadoop cluster A or Hadoop cluster B is faulty. Services are provided to the outside world in the form of a unified proxy. Based on the value of serverState, the unified proxy constructs the cluster access address without the client's awareness and directs it to the cluster that actually performs the storage and computing tasks. Compared with a single Hive cluster, this greatly improves the continuity, availability and reliability of accessing the required business and tasks.

[0090] See Figure 4 As shown, when the primary cluster fails (i.e., cluster1 fails) and the backup cluster is normal (i.e., cluster2 is normal), the handling method is as follows: When the value of serverState of cluster2 is equal to 0, the data disaster recovery direction is cluster1->cluster2, and the Hive metadata of Hadoop cluster A and Hadoop cluster B is synchronized in real time, including: periodic T triggering the synchronization of incremental database table data (HDFS incremental file); triggering after the execution of the storage and compute scheduling task, when the incremental database table data is greater than M, HDFS incremental file synchronization is performed. The size of M can be set by those skilled in the art according to business and disaster recovery requirements. In this embodiment, M is the value of the network bandwidth S between cluster1 and cluster2 multiplied by 4 hours. The following M is uniformly adopted; when the value of serverState of cluster2 is equal to 1, the data disaster recovery direction is cluster1->cluster2, and the Hive metadata of Hadoop cluster A and Hadoop cluster B is synchronized in real time.

[0091] For further details, please refer to [link / reference]. Figure 5As shown, when cluster1 recovers from a failure and cluster2 remains normal, the data disaster recovery direction is cluster2->cluster1. The processing methods are divided into two types: executing based on the failure of the cluster1 metadata database and using Canal to execute the Hive metadata synchronization between cluster1 and cluster2.

[0092] S200: If the cluster1 metadata database is normal, proceed directly to S201. If the cluster1 metadata database fails, obtain the timestamp of the latest data in the BinLog of the MySQL metadata database of cluster1 and cluster2 at the current moment, extract the historical changes of metadata of cluster2 during the failure of cluster1 according to the timestamp, and perform synchronization on cluster1.

[0093] S201. Using Hive's metadata / MySQL's binlog, parse out the incremental information of Hive database table data during the period from cluster1 failure to recovery;

[0094] S202. Use the distcp script to synchronize incremental information to cluster1;

[0095] S203, Data consistency check. If it fails, return to S202 loop. If it succeeds, proceed to S206 (database table partition, file size, number of data records, etc.).

[0096] S204. Using Canal to perform Hive metadata synchronization between cluster1 and cluster2 is divided into:

[0097] Period T triggers the synchronization of data in the incremental database table (HDF incremental file);

[0098] Triggered upon completion of the weekly storage and computation task, when the incremental database table data exceeds M, HDFS incremental file synchronization is executed;

[0099] S205, Data consistency check (database table partitions, file size, number of records, etc.). If it fails, return to S204 and loop. If it passes, proceed to S206 to execute.

[0100] S206, Data consistency verification during the start and recovery times of cluster1 failure;

[0101] S207 If the verification passes, the cluster service can be switched from cluster2 to cluster1. If the verification fails, return to S205 or S203 in a loop.

[0102] Furthermore, see Figure 6As shown, if cluster1 remains normal and cluster2 recovers from a fault, the data disaster recovery direction is cluster1->cluster2. The processing methods are divided into two types: executing based on the cluster2 metadata database failure and using Canal to execute Hive metadata synchronization between cluster1 and cluster2.

[0103] S300: If the cluster1 metadata database is normal, proceed directly to S301. If the cluster2 metadata database fails, obtain the timestamp of the latest data in the BinLog of the MySQL metadata database of cluster1 and cluster2 at the current moment, extract the historical changes of metadata of cluster1 during the failure of cluster2 according to the timestamp, and perform synchronization on cluster2.

[0104] S301. Using Hive's metadata / MySQL's binlog, parse out the incremental information of Hive database table data during the period from cluster2 failure to recovery;

[0105] S302. Use the distcp script to synchronize incremental information to cluster2;

[0106] S303, Data consistency check. If it fails, return to S302 loop. If it succeeds, proceed to S306 (database table partition, file size, number of data records, etc.).

[0107] S304. Using Canal to perform Hive metadata synchronization between cluster1 and cluster2 is divided into:

[0108] Period T triggers the synchronization of data in the incremental database table (HDF incremental file);

[0109] Triggered upon completion of the weekly storage and computation task, when the incremental database table data exceeds M, HDFS incremental file synchronization is executed;

[0110] S305, Data consistency check (database table partitions, file size, number of records, etc.). If it fails, return to S304 and loop. If it passes, proceed to S306 to execute.

[0111] S306, during the start and recovery times of cluster2 failure, data consistency is checked. If the check passes, the process ends; otherwise, it returns to the S305 or S303 loop.

[0112] A data and task disaster recovery system that spans multiple Hive clusters is used to implement the aforementioned data and task disaster recovery method that spans multiple Hive clusters. It includes a cluster monitoring Listener module and a unified proxy module, and is deployed on a cloud with high availability and security, outside of cluster1 and cluster2.

[0113] The cluster monitoring Listener module sends test scripts at time intervals of T (in this embodiment, fault detection is based on minutes, and the initial T is set to 1 minute). It then traverses the metadata database, Hive database, and HDFS cluster of test cluster1 and cluster2 to obtain the health status of the two clusters and serverState. T is an integer greater than or equal to 1.

[0114] If a failure is detected in cluster1 or cluster2, data disaster recovery will be performed, i.e., according to... Figure 5 and Figure 6 The process shown is executed, and the unified proxy module is modified to replace the task scheduling cluster.

[0115] It is known that the deployment of datanodes in the HDFS cluster on which Hive depends adopts rack awareness to ensure that at least one replica node is on a different rack. If the cluster monitoring Listener module detects that 1 / 3 of the datanode nodes are faulty, then cluster1 and cluster2 will be switched, and data disaster recovery and task scheduling will be switched accordingly.

[0116] In response to a failure of cluster1 or cluster2, the cluster monitoring Listener module performs cluster health status detection according to the incrementing failure period T = T + K * detection result return time, where K is a constant between (0,1) and T ≤ 1h; if cluster1 or cluster2 recovers to normal, T is set to the initial value, which is 1 minute as set in this embodiment.

[0117] It is worth mentioning that when there are more than two clusters that serve as disaster recovery for each other, that is, in addition to cluster1 and cluster2 there is clusterN, when the primary cluster fails and recovers, an election mechanism can be used to select the cluster with the most and most up-to-date metadata and the most HDFS file data among all clusters as the primary cluster and reference cluster for task scheduling and data recovery.

[0118] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0119] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for data and task disaster recovery across multiple Hive clusters, characterized in that, Including the following steps: Step 1: Obtain Kerberos authentication for Hadoop cluster A and Hadoop cluster B, establish cross-domain trust, and synchronize tenant information. Step 2: Obtain data from Hadoop cluster A and Hadoop cluster B, and copy and synchronize them; use a data replication tool to replicate and synchronize data between the two clusters A and B to maintain data consistency. The process of replicating and synchronizing cluster data between Hadoop cluster A and Hadoop cluster B includes: obtaining the cluster health status (serverState); using the canal tool to perform real-time synchronization of Hive metadata in the MySQL metadata database of the two Hive clusters; verifying the Hive metadata synchronization results; if the verification fails, an error alarm is generated and the process returns to execute real-time Hive metadata synchronization; if the verification is successful, the distcp script tool is used to perform HDFS data synchronization; and verifying the data consistency of the HDFS synchronization results; if the verification fails, an error alarm is generated and the process returns to execute HDFS data synchronization; if the verification is successful, the data synchronization between the two Hive clusters is considered complete. Step 3: Data loss or corruption in Hadoop cluster A, receiving data scheduling from Hadoop cluster B; Data loss or corruption in Hadoop cluster B, receiving data scheduling from Hadoop cluster A; Task scheduling: Using a distributed task scheduling system to manage and schedule tasks, ensuring disaster recovery switching between clusters A and B. Step 4: If data in Hadoop cluster A is lost or corrupted, receive data from Hadoop cluster B to repair the data in Hadoop cluster A. If data in Hadoop cluster B is lost or corrupted, receive data from Hadoop cluster A to repair the data in Hadoop cluster B. Failover: Configure a failover mechanism so that when cluster A fails, it automatically switches to cluster B to perform tasks and restores data synchronization in a timely manner.

2. The method for data and task disaster recovery across multiple Hive clusters according to claim 1, characterized in that, The method for synchronizing data between Hadoop cluster A and Hadoop cluster B includes the following steps: The server disk hosting the metadata database of the Hive cluster spanning multiple data centers / regions is configured as RAID 1 / RAID 10.

3. The method for data and task disaster recovery across multiple Hive clusters according to claim 1, characterized in that, The method for verifying the Hive metadata synchronization result includes: Obtain the table structure and metadata, database objects, table data statistics, partition information, and Hive configuration information of Hadoop cluster A and Hadoop cluster B respectively, and compare their consistency. If they match, it is considered normal; If there is a discrepancy, an error alert will be generated.

4. The method for data and task disaster recovery across multiple Hive clusters according to claim 1, characterized in that, The method for verifying the consistency of HDFS synchronization results is as follows: obtain the file lists and file contents of Hadoop cluster A and Hadoop cluster B respectively; Compare the file lists of Hadoop cluster A and Hadoop cluster B to obtain the file lists on the source HDFS and target HDFS, including file paths, file sizes, and modification times. By comparing the file lists on the two clusters, the consistency of files between the source and target is ensured. Compare the hash values ​​of the file contents in Hadoop cluster A and Hadoop cluster B. If the hash values ​​are the same, the file contents are consistent; otherwise, they are inconsistent.

5. The method for data and task disaster recovery across multiple Hive clusters according to claim 1, characterized in that, The method for monitoring failures in Hadoop cluster A and Hadoop cluster B includes the following steps: Get the serverState values ​​of Hadoop cluster A and Hadoop cluster B respectively. Hadoop cluster A is marked as cluster1 and Hadoop cluster B is marked as cluster2. Cluster1 serves as the master data service and is normally the one that provides the service. serverState=0 means that both cluster1 and cluster2 are providing normal service; serverState=1 means that only cluster1 is faulty; serverState=2 means that only cluster2 is faulty; serverState=3 indicates that both cluster1 and cluster2 are down.

6. The method for data and task disaster recovery across multiple Hive clusters according to claim 5, characterized in that, When the value of serverState equals 0, the data disaster recovery direction is cluster1 -> cluster2, and the Hive metadata of Hadoop cluster A and Hadoop cluster B is synchronized in real time, including: Period T triggers incremental data synchronization between database tables. Triggered upon completion of the storage and computing scheduling task, when the incremental database table data is greater than M, HDFS incremental file synchronization is executed, where M is the value of network bandwidth S between cluster1 and cluster2 multiplied by 4 hours. When the value of serverState is equal to 1, the data disaster recovery direction is cluster2-> cluster1, and the Hive metadata of Hadoop cluster A and Hadoop cluster B are synchronized in real time.

7. A method for data and task disaster recovery across multiple Hive clusters according to claim 6, characterized in that, When cluster1 recovers from a failure and cluster2 remains normal, the data disaster recovery direction is cluster2-> cluster1. The handling methods are divided into two types: executing based on the failure of the cluster1 metadata database and using Canal to perform Hive metadata synchronization between cluster1 and cluster2. S200: If the cluster1 metadata database is normal, proceed directly to S201. If the cluster1 metadata database fails, obtain the timestamp of the latest data in the binlog of the mysql metadata databases of cluster1 and cluster2 at the current moment, extract the historical changes of metadata of cluster2 during the failure of cluster1 according to the timestamp, perform synchronization on cluster1, and execute S201. S201. Using Hive's metadata / MySQL's binlog, parse out the incremental information of Hive database table data during the period from cluster1 failure to recovery; S202. Use the distcp script to synchronize incremental information to cluster1; S203, Data consistency check. If it fails, return to S202 and loop. If it succeeds, proceed to S206. S204. Using Canal to perform Hive metadata synchronization between cluster1 and cluster2 is divided into: Period T triggers data synchronization for the newly added database table; Triggered upon completion of the weekly storage and computation task, when the incremental database table data exceeds M, HDFS incremental file synchronization is executed; S205, Data consistency check. If it fails, return to S204 and loop. If it succeeds, proceed to S206 to execute. S206, Data consistency verification during the start and recovery times of cluster1 failure; S207 If the verification passes, the cluster service switches from cluster2 to cluster1. If the verification fails, it returns to S205 or S203 in a loop.

8. A method for data and task disaster recovery across multiple Hive clusters according to claim 5, characterized in that, When cluster1 is always normal and cluster2 recovers from the failure, the data disaster recovery direction is cluster1-> cluster2. The handling methods are divided into two types: executing based on the failure of the cluster2 metadata database and using Canal to perform Hive metadata synchronization between cluster1 and cluster2. S300: If the cluster2 metadata database is normal, proceed directly to S301. If the cluster2 metadata database fails, obtain the timestamp of the latest data in the binlog of the cluster1 and cluster2 metadata databases MySQL at the current moment, extract the historical changes of metadata of cluster1 during the cluster2 failure according to the timestamp, perform synchronization on cluster2, and execute S301. If the cluster2 metadata database is normal, then execute S304. S301. Using Hive's metadata / MySQL's binlog, parse out the incremental information of Hive database table data during the period from cluster2 failure to recovery; S302. Use the distcp script to synchronize incremental information to cluster2; S303, Data consistency check. If it fails, return to S302 loop; if it succeeds, proceed to S306. S304. Using Canal to perform Hive metadata synchronization between cluster1 and cluster2 is divided into: Period T triggers data synchronization for the newly added database table; Triggered upon completion of the weekly storage and computation task, when the incremental database table data exceeds M, HDFS incremental file synchronization is executed; S305, Data consistency check. If it fails, return to S304 and loop. If it succeeds, proceed to S306 to execute. S306, during the start and recovery times of cluster2 failure, data consistency is checked. If the check passes, the process ends; otherwise, it returns to the S305 or S303 loop.

9. A data and task disaster recovery system across multiple Hive clusters, used to implement the data and task disaster recovery method across multiple Hive clusters as described in any one of claims 1-8, characterized in that, It includes a cluster monitoring Listener module and a unified proxy module, deployed on a cloud with high availability and security, outside of cluster1 and cluster2; Among them, Hadoop cluster A is labeled as cluster1, Hadoop cluster B is labeled as cluster2, and cluster1 serves as the master data service; The cluster monitoring Listener module sends test scripts at time intervals of T, traverses the metadata database, Hive database, and HDFS cluster of test cluster1 and cluster2, and obtains the health status of the two clusters and serverState, where T is an integer greater than or equal to 1. serverState=0 means that both cluster1 and cluster2 are functioning normally; serverState=1 means that only cluster1 is failing; serverState=2 means that only cluster2 is failing; serverState=3 means that both cluster1 and cluster2 are failing. If a failure is detected in cluster1 or cluster2, data disaster recovery will be performed, and the task scheduling cluster will be replaced by a unified proxy module.

10. A data and task disaster recovery system across multiple Hive clusters as described in claim 9, characterized in that, If the cluster monitoring Listener module detects a failure in 1 / 3 of the DataNode nodes in cluster1 or cluster2, it will switch between cluster1 and cluster2, and correspondingly switch data disaster recovery and task scheduling. In the event of a failure in cluster1 or cluster2, the cluster monitoring Listener module performs cluster health status detection according to the incrementing failure period T = T + K * detection result return time, where K is a constant between (0,1) and T ≤ 1h; if cluster1 or cluster2 recovers to normal, T is set to the initial value.

Images