Hive table data backup and recovery method

Through the Tomcat application and Hive Client combined with the Kerberos environment, Hive table data backup and recovery is realized, solving the problems of increased business volume and insufficient data security in the existing technology, and improving the stability and data security of the system.

CN119025337APending Publication Date: 2024-11-26SHANGHAI INSPUR CLOUD COMPUTING SERVICE CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411088345.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing Hive table data backup and recovery methods are difficult to cope with the increase in business volume, and there are shortcomings in data security, which requires improving the stability and security of the system.

Method used

The Tomcat application and Hive Client are combined with the Kerberos environment to realize the backup and recovery of Hive table data. The Tomcat application front-end sends requests, parses parameters, obtains Kerberos tickets, connects Zookeeper and HDFS Namenode, performs data backup and recovery operations, and updates metadatabase information.

Benefits of technology

It improves the controllability and ease of use of operations, reduces manual intervention and operational errors, improves operation efficiency and accuracy, and improves system stability and data security.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025337A_ABST
    Figure CN119025337A_ABST
Patent Text Reader

Abstract

The invention discloses a Hive table data backup and recovery method, and relates to the technical field of data management. Comprising the following steps: a Tomcat application front end sends a Hive table data backup or recovery request to a Tomcat application rear end; the back end of the Tomcat application analyzes related request parameters according to the received different requests; connecting to a server where the Tomcat application is located, and acquiring a Kerberos bill of the Hive user; a Kerberos bill is used for being connected with a Zookeeper node, and Hive cluster information in a Hive ZNode is obtained, and the Hive cluster information comprises a node where a Hive Server 2 is located, a node where a Hive Metacore is located and the like; the method comprises the following steps: connecting a Hive Metacore, and obtaining a storage path of a Hive library table specified by a user in an HDFS and metadata of the table; if the data backup is the Hive table data backup, connecting an HDFS Namenode node by using a Kerberos bill, and backing up a file under an HDFS path corresponding to the Hive table to a position specified by a user in a corresponding format; if the data of the Hive table is recovered, connecting to a server specified by a user to verify the validity of the backup file, then calling a Hive Client API (Application Program Interface) by using a Kerberos bill, and recovering the backup file into the Hive table; related backup recovery data information is updated in the back-end data table of the Tomcat application, and the front end of the Tomcat application judges whether the backup recovery task is executed successfully or not by querying the back-end database table.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention discloses a Hive table data backup and recovery method, and relates to the technical field of data management. Background Art

[0002] HDFS (Hadoop Distributed File System) is one of the core components of the Apache Hadoop ecosystem, used to store and process large-scale data sets. The working principle of HDFS is to split large files into small blocks and store these blocks on multiple nodes in the cluster to achieve distributed storage and processing of data.

[0003] Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation and loading. It is a mechanism for storing, querying and analyzing large-scale data stored in Hadoop. The Hive data warehouse tool can map structured data files to a database table and provide SQL query functions, which can convert SQL statements into MapReduce tasks for execution. Hive data backup is crucial for protecting and managing large-scale data. However, the current backup and recovery methods cannot cope with the increase in business volume, and data security also needs to be strengthened. Summary of the invention

[0004] In view of the problems of the prior art, the present invention provides a Hive table data backup and recovery method, which implements Hive table data backup and recovery in a Kerberos environment based on Tomcat application and Hive Client, thereby increasing business stability and data security.

[0005] The specific scheme proposed by the present invention is:

[0006] The present invention provides a Hive table data backup and recovery method, comprising:

[0007] Step 1: Send a Hive table data backup or restore request to the Tomcat application backend based on the Tomcat application frontend;

[0008] Step 2: Parse the relevant request parameters based on the received request through the Tomcat application backend;

[0009] Step 3: Connect to the server where the Tomcat application is located through the Tomcat application backend to obtain the Hive user Kerberos ticket;

[0010] Step 4: Use the Kerberos ticket to connect to the Zookeeper node through the Tomcat application backend to obtain the Hive cluster information in HiveZNode. The Hive cluster information includes the node where Hive Server2 is located and the node where Hive Metastore is located.

[0011] Step 5: Connect to Hive Metastore through the Tomcat application backend to obtain the storage path of the user-specified Hive table in HDFS and the metadata of the table. The metadata is used for data analysis and reconstruction during the Hive table data backup or recovery process;

[0012] Step 6: Obtain the request type based on the metadata: If the request type is Hive table data backup, use the Kerberos ticket to connect to the HDFS Namenode node, and back up the files under the HDFS storage path corresponding to the Hive table in the corresponding format to the user-specified location; If the request type is Hive table data recovery, connect to the user-specified server to verify the validity of the backup file, then use the Kerberos ticket to call the Hive Client API, restore the backup file to the Hive table in the specified format, and update the Hive metadata database information;

[0013] Step 7: Update the relevant backup and recovery data information in the Tomcat application backend data table, and query the backend database table through the Tomcat application front end to determine whether the backup and recovery task is successfully executed.

[0014] Furthermore, in step 1 of the Hive table data backup and recovery method, the Tomcat application front end is managed by the Tomcat server, options for backing up and restoring Hive table data are provided, the required operation type is selected in the Tomcat application front end, the operation type includes backing up Hive library tables and restoring Hive library tables, and related parameters for backup or recovery are specified, and the selected operation type and related parameters are encapsulated into an HTTP request through the Tomcat application front end and sent to the Tomcat application back end.

[0015] Furthermore, in step 2 of the Hive table data backup and recovery method, the Spring MVC framework is used to parse the request through the Tomcat application backend, extract the operation type and related parameters in the request, and classify and verify all required parameters in the related parameters to ensure that all required parameters conform to the expected format and application logic. If the parameter verification is incorrect, the error information is promptly fed back to the user, and the backup or recovery operation is terminated.

[0016] Furthermore, in step 3 of the Hive table data backup and recovery method, the Kerberos ticket of the Hive user is obtained through the Tomcat application backend to ensure that subsequent operations pass the Kerberos authentication. After that, the Tomcat application backend interacts with the Kerberos KDC to obtain the TGT of the Hive user, and the TGT of the Hive user is used as the Kerberos credential for all authentication requests.

[0017] After obtaining the Hive user's TGT, request a service ticket from the KDC through the Tomcat application backend for authentication and authorization with Hive and other services.

[0018] In step 4, the Zookeeper Java API is used to connect to the Zookeeper cluster through the Tomcat application backend, and the Kerberos ticket is used to interact with Zookeeper to obtain the Hive cluster information. The ZNode data is read through the Tomcat application backend: Hive natively uses Zookeeper to store cluster metadata information, including the address of HiveServer2 and the location of HiveMetastore.

[0019] Parse Zookeeper data through the Tomat application backend: read the ZNode data and extract specific configuration information, including the URL of Hive Server2 and the IP address and port of Metastore.

[0020] Furthermore, in step 6 of the Hive table data backup and recovery method, if the request type is Hive table data backup, the method specifically includes:

[0021] Use the HDFS Java API to connect to the HDFS Namenode using the Kerberos ticket through the Tomcat application backend.

[0022] Data replication: Copy all files in the Hive table storage path on HDFS to the backup location specified by the user.

[0023] Backup format selection: Select the backup file format based on user needs and system requirements. The backup file formats include HDFS file format, compressed file format, and snapshot format.

[0024] Furthermore, in step 6 of the Hive table data backup and recovery method, if the request type is Hive table data recovery, the method specifically includes:

[0025] Verify the validity of the backup file by connecting to the user-specified server through the Tomcat application backend. Verifying the validity of the backup file includes checking the integrity of the file, checking whether the file format is correct, and checking whether the file contains complete data.

[0026] Use the Kerberos ticket to call the Hive Client API to connect to HiveServer2 through the Tomcat application backend, restore the backup file to the Hive table in the specified style, and update the Hive metadata database information, including reloading the table partition information and recalculating the statistics.

[0027] Further, in step 7 of the Hive table data backup and recovery method, updating the relevant backup and recovery data information in the Tomcat application backend data table includes:

[0028] Record task status: Record the status information of each backup or restore task in the database of the Tomcat application backend, including the operation type, operation object, execution time, and result status.

[0029] Record logs: Record operation logs in detail for subsequent audits and troubleshooting. Operation logs include operation steps, execution time, and exception information.

[0030] The present invention also provides a Hive table data backup and recovery device, including a Tomcat front-end management block, a Tomcat back-end management module and a backup and recovery management module.

[0031] The Tomcat application front end uses the mcat front end management block to send Hive table data backup or recovery requests to the Tomcat application back end;

[0032] The Tomcat application backend parses the relevant request parameters based on the received request through the Tomcat backend management module;

[0033] The Tomcat application backend connects to the server where the Tomcat application is located through the Tomcat backend management module to obtain the Hive user Kerberos ticket;

[0034] The Tomcat application backend uses the Kerberos ticket to connect to the Zookeeper node through the Tomcat backend management module and obtain the Hive cluster information in the Hive ZNode. The Hive cluster information includes the node where Hive Server2 is located and the node where HiveMetastore is located.

[0035] The Tomcat application backend connects to the Hive Metastore through the Tomcat backend management module to obtain the storage path of the Hive library table specified by the user in HDFS and the metadata of the table. The metadata is used for data analysis and reconstruction during the Hive table data backup or recovery process;

[0036] The backup and recovery management module obtains the request type based on the metadata: if the request type is Hive table data backup, the Kerberos ticket is used to connect to the HDFS Namenode node, and the files under the HDFS storage path corresponding to the Hive table are backed up to the user-specified location in the corresponding format; if the request type is Hive table data recovery, the module connects to the user-specified server to verify the validity of the backup file, and then uses the Kerberos ticket to call the Hive Client API to restore the backup file to the Hive table in the specified format, and update the Hive metadata database information;

[0037] The Tomcat backend management module updates the relevant backup and recovery data information in the Tomcat application backend data table, and determines whether the backup and recovery task is successfully executed by querying the backend database table through the Tomcat application frontend.

[0038] The benefits of the present invention are:

[0039] The present invention implements Hive table data backup and recovery in Kerberos environment based on Tomcat application and Hive Client. Compared with the commonly used backup and recovery methods, this solution has the following innovations:

[0040] First, the Tomcat application is used as the front end to receive user requests, and the backup and recovery operations are incorporated into a unified management framework, which improves the controllability and usability of the operation. In the prior art, backup and recovery operations often need to be performed through command lines or special management tools. However, the solution of the present invention uses the Tomcat application front end, allowing users to complete operation requests through a simple interface, lowering the operation threshold and improving the user experience.

[0041] Secondly, by using Tomcat application backend to parse parameters, cache Kerberos tickets, connect to Hive clusters, etc., the automatic management of Hive table data backup and recovery operations is realized, reducing the possibility of manual intervention and operational errors. Compared with the existing manual operation method, this solution greatly improves the efficiency and accuracy of operations, reduces operational risks, and improves the stability and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0044] Example 1

[0045] The present invention provides a Hive table data backup and recovery method, comprising:

[0046] Step 1: Send a Hive table data backup or restore request to the Tomcat application backend based on the Tomcat application frontend.

[0047] In step 1, you can further manage the Tomcat application front end through the Tomcat server.

[0048] Provides options for backing up and restoring Hive table data. Select the required operation type on the Tomcat application front end, which includes backing up Hive tables and restoring Hive tables. Specify the backup or restore parameters. The Tomcat application front end encapsulates the selected operation type and related parameters, such as table name and backup path, into an HTTP request and sends it to the Tomcat application back end.

[0049] Step 2: Parse the relevant request parameters based on the received request through the Tomcat application backend. The Tomcat application backend uses the Spring MVC framework to parse the request, extract the operation type and related parameters in the request, and classify and verify all required parameters in the related parameters to ensure that all required parameters conform to the expected format and application logic, such as verifying the legitimacy of the Hive table name, checking the writability of the backup path, or checking the readability of the recovery file path.

[0050] If the parameter verification is incorrect, the error information is fed back to the user in a timely manner, and the backup or recovery operation is terminated. For example, if the parameter is found to be incorrect during the parsing process, the error information is fed back to the user in a timely manner, and the backup or recovery operation is terminated. The Tomcat application backend notifies the front-end application to make corresponding feedback by returning a specific HTTP status code and error message, reminding the user to check the reason for the operation failure.

[0051] Step 3: Connect to the server where the Tomcat application is located through the Tomcat application backend to obtain the Hive user Kerberos ticket.

[0052] The Kerberos ticket of the Hive user is obtained through the Tomcat application backend to ensure that subsequent operations pass Kerberos authentication. Then, the Tomcat application backend interacts with the Kerberos KDC (Key Distribution Center) to obtain the TGT (Ticket Granting Ticket) of the Hive user, and the TGT of the Hive user is used as the Kerberos credential for all authentication requests.

[0053] After obtaining the Hive user's TGT, request a service ticket (ServiceTicket) from KDC through the Tomcat application backend for authentication and authorization with Hive and other services. To improve efficiency and reduce the overhead of frequent authentication, the ticket can be cached for a fixed time. When obtaining Kerberos credentials on the Tomcat application backend, ensure the security of the credentials to avoid leakage. This solution can enhance security by setting a reasonable ticket validity period and refresh policy.

[0054] Step 4: Use the Kerberos ticket to connect to the Zookeeper node through the Tomcat application backend to obtain the Hive cluster information in HiveZNode. The Hive cluster information includes the node where Hive Server2 is located and the node where Hive Metastore is located.

[0055] The Zookeeper Java API is used to connect to the Zookeeper cluster through the Tomcat application backend, and the Kerberos ticket is used to interact with Zookeeper to obtain Hive cluster information. The ZNode data is read through the Tomcat application backend: Hive natively uses Zookeeper to store cluster metadata information, including the address of HiveServer2 and the location of HiveMetastore.

[0056] Parse Zookeeper data through the Tomat application backend: read the ZNode data and extract specific configuration information, including the URL of Hive Server2 and the IP address and port of Metastore.

[0057] Step 5: Connect to Hive Metastore through the Tomcat application backend to obtain the storage path of the user-specified Hive library table in HDFS and the metadata of the table. The metadata is used for data analysis and reconstruction during the Hive table data backup or recovery process.

[0058] The Hive Metastore API is used to query the metadata information of the specified Hive table, which includes the table column information, partition information, etc. The metadata is used for data analysis and reconstruction during the Hive table data backup and recovery process. Finally, the storage path of the table on HDFS is queried to obtain the physical storage location of the Hive table data as the target path during data recovery and backup.

[0059] Step 6: Obtain the request type based on the metadata: If the request type is Hive table data backup, use the Kerberos ticket to connect to the HDFS Namenode node, and back up the files under the HDFS storage path corresponding to the Hive table in the corresponding format to the user-specified location; if the request type is Hive table data recovery, connect to the user-specified server to verify the validity of the backup file, and then use the Kerberos ticket to call the Hive Client API to restore the backup file to the Hive table in the specified format and update the Hive metadata database information.

[0060] If the request type is Hive table data backup, it specifically includes:

[0061] Use the HDFS Java API to connect to the HDFS Namenode using the Kerberos ticket through the Tomcat application backend.

[0062] Data replication: Copy all files in the Hive table storage path on HDFS to the backup location specified by the user.

[0063] Backup format selection: Select the backup file format according to user needs and system requirements. The backup file formats include HDFS file format, compressed file formats such as Zip, Tar.gz, and snapshot formats. Users can choose the appropriate backup format to improve the efficiency of backup and recovery.

[0064] If the request type is Hive table data recovery, it includes:

[0065] Verify the validity of the backup file by connecting to the user-specified server through the Tomcat application backend. Verifying the validity of the backup file includes checking the integrity of the file, checking whether the file format is correct, and checking whether the file contains complete data.

[0066] Use the Kerberos ticket to call the Hive Client API to connect to HiveServer2 through the Tomcat application backend, restore the backup file to the Hive table in the specified style, and update the Hive metadata database information, including reloading the table partition information and recalculating the statistics.

[0067] Step 7: Update the relevant backup and recovery data information in the Tomcat application backend data table, and query the backend database table through the Tomcat application front end to determine whether the backup and recovery task is successfully executed.

[0068] The relevant backup and recovery data information is updated in the Tomcat application backend data table, including:

[0069] Record task status: Record the status information of each backup or restore task in the database of the Tomcat application backend, including the operation type, operation object, execution time, and result status.

[0070] Record logs: Record operation logs in detail for subsequent audits and troubleshooting. Operation logs include operation steps, execution time, and exception information.

[0071] When querying results, users can query the execution status and results of the task through the front-end interface to understand whether the backup or recovery operation is successful, as well as detailed operation logs or error information.

[0072] Example 2

[0073] The present invention also provides a Hive table data backup and recovery device, including a Tomcat front-end management block, a Tomcat back-end management module and a backup and recovery management module.

[0074] The Tomcat application front end uses the mcat front end management block to send Hive table data backup or recovery requests to the Tomcat application back end;

[0075] The Tomcat application backend parses the relevant request parameters based on the received request through the Tomcat backend management module;

[0076] The Tomcat application backend connects to the server where the Tomcat application is located through the Tomcat backend management module to obtain the Hive user Kerberos ticket;

[0077] The Tomcat application backend uses the Kerberos ticket to connect to the Zookeeper node through the Tomcat backend management module and obtain the Hive cluster information in the Hive ZNode. The Hive cluster information includes the node where Hive Server2 is located and the node where HiveMetastore is located.

[0078] The Tomcat application backend connects to the Hive Metastore through the Tomcat backend management module to obtain the storage path of the Hive library table specified by the user in HDFS and the metadata of the table. The metadata is used for data analysis and reconstruction during the Hive table data backup or recovery process;

[0079] The backup and recovery management module obtains the request type based on the metadata: if the request type is Hive table data backup, the Kerberos ticket is used to connect to the HDFS Namenode node, and the files under the HDFS storage path corresponding to the Hive table are backed up to the user-specified location in the corresponding format; if the request type is Hive table data recovery, the module connects to the user-specified server to verify the validity of the backup file, and then uses the Kerberos ticket to call the Hive Client API to restore the backup file to the Hive table in the specified format, and update the Hive metadata database information;

[0080] The Tomcat backend management module updates the relevant backup and recovery data information in the Tomcat application backend data table, and determines whether the backup and recovery task is successfully executed by querying the backend database table through the Tomcat application frontend.

[0081] The information interaction of the processor in the above-mentioned device, the execution process of the readable program, etc., are based on the same concept as the embodiment of the method of the present invention. The specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0082] Similarly, compared with the commonly used backup and recovery methods, the device of the present invention has the following advantages:

[0083] First, the Tomcat application is used as the front end to receive user requests, and the backup and recovery operations are incorporated into a unified management framework, which improves the controllability and usability of the operation. In the prior art, backup and recovery operations often need to be performed through command lines or special management tools. However, the solution of the present invention uses the Tomcat application front end, allowing users to complete operation requests through a simple interface, lowering the operation threshold and improving the user experience.

[0084] Secondly, by using Tomcat application backend to parse parameters, cache Kerberos tickets, connect to Hive clusters, etc., the automatic management of Hive table data backup and recovery operations is realized, reducing the possibility of manual intervention and operational errors. Compared with the existing manual operation method, this solution greatly improves the efficiency and accuracy of operations, reduces operational risks, and improves the stability and reliability of the system.

[0085] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0086] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A Hive table data backup and recovery method, characterized by: include: Step 1: Send a Hive table data backup or restore request to the Tomcat application backend based on the Tomcat application frontend; Step 2: Parse the relevant request parameters based on the received request through the Tomcat application backend; Step 3: Connect to the server where the Tomcat application is located through the Tomcat application backend to obtain the Hive user Kerberos ticket; Step 4: Use the Kerberos ticket to connect to the Zookeeper node through the Tomcat application backend to obtain the Hive cluster information in the Hive ZNode. The Hive cluster information includes the node where Hive Server2 is located and the node where Hive Metastore is located; Step 5: Connect to Hive Metastore through the Tomcat application backend to obtain the storage path of the user-specified Hive table in HDFS and the metadata of the table. The metadata is used for data analysis and reconstruction during the Hive table data backup or recovery process; Step 6: Obtain the request type based on the metadata: If the request type is Hive table data backup, use the Kerberos ticket to connect to the HDFS Namenode node, and back up the files under the HDFS storage path corresponding to the Hive table in the corresponding format to the user-specified location; If the request type is Hive table data recovery, connect to the user-specified server to verify the validity of the backup file, then use the Kerberos ticket to call the Hive Client API, restore the backup file to the Hive table in the specified format, and update the Hive metadata database information; Step 7: Update the relevant backup and recovery data information in the Tomcat application backend data table, and query the backend database table through the Tomcat application front end to determine whether the backup and recovery task is successfully executed.

2. A Hive table data backup and recovery method according to claim 1, characterized in that In step 1, the Tomcat application front end is managed through the Tomcat server, and options for backing up and restoring Hive table data are provided. The required operation type is selected on the Tomcat application front end, including backing up Hive library tables and restoring Hive library tables, and related parameters for backup or restoration are specified. The Tomcat application front end encapsulates the selected operation type and related parameters into an HTTP request and sends it to the Tomcat application back end.

3. A Hive table data backup and recovery method according to claim 1, characterized in that In step 2, the Spring MVC framework is used to parse the request through the Tomcat application backend, extract the operation type and related parameters in the request, and classify and verify all required parameters in the related parameters to ensure that all required parameters conform to the expected format and application logic. If the parameter verification is incorrect, the error information is fed back to the user in a timely manner and the backup or recovery operation is terminated.

4. A Hive table data backup and recovery method according to claim 1, characterized in that In step 3, the Kerberos ticket of the Hive user is obtained through the Tomcat application backend to ensure that subsequent operations pass Kerberos authentication. Then, the Tomcat application backend interacts with the Kerberos KDC to obtain the TGT of the Hive user, and the TGT of the Hive user is used as the Kerberos credential for all authentication requests. After obtaining the Hive user's TGT, request a service ticket from the KDC through the Tomcat application backend for authentication and authorization with Hive and other services. In step 4, the Zookeeper Java API is used to connect to the Zookeeper cluster through the Tomcat application backend, and the Kerberos ticket is used to interact with Zookeeper to obtain the Hive cluster information. The ZNode data is read through the Tomcat application backend: Hive natively uses Zookeeper to store cluster metadata information, including the address of HiveServer2 and the location of HiveMetastore. Parse Zookeeper data through the Tomat application backend: read the ZNode data and extract specific configuration information, including the URL of Hive Server2 and the IP address and port of Metastore.

5. A Hive table data backup and recovery method according to claim 1, characterized in that If the request type in step 6 is Hive table data backup, the steps specifically include: Use the HDFS Java API to connect to the HDFS Namenode using the Kerberos ticket through the Tomcat application backend. Data replication: Copy all files in the Hive table storage path on HDFS to the backup location specified by the user. Backup format selection: Select the backup file format based on user needs and system requirements. The backup file formats include HDFS file format, compressed file format, and snapshot format.

6. A Hive table data backup and recovery method according to claim 1, characterized in that If the request type in step 6 is Hive table data recovery, the steps specifically include: Verify the validity of the backup file by connecting to the user-specified server through the Tomcat application backend. Verifying the validity of the backup file includes checking the integrity of the file, checking whether the file format is correct, and checking whether the file contains complete data. Use the Kerberos ticket to call the Hive Client API through the Tomcat application backend to connect to Hive Server2, restore the backup file to the Hive table in the specified style, and update the Hive metadata database information, including reloading the table partition information and recalculating the statistics.

7. A Hive table data backup and recovery method according to claim 1, characterized in that In step 7, the relevant backup and recovery data information is updated in the Tomcat application backend data table, including: Record task status: Record the status information of each backup or restore task in the database of the Tomcat application backend, including the operation type, operation object, execution time, and result status. Record logs: Record operation logs in detail for subsequent audits and troubleshooting. Operation logs include operation steps, execution time, and exception information.

8. A Hive table data backup and recovery device, characterized in that Including Tomcat front-end management module, Tomcat back-end management module and backup and recovery management module. The Tomcat application front end uses the mcat front end management block to send Hive table data backup or recovery requests to the Tomcat application back end; The Tomcat application backend parses the relevant request parameters based on the received request through the Tomcat backend management module; The Tomcat application backend connects to the server where the Tomcat application is located through the Tomcat backend management module to obtain the Hive user Kerberos ticket; The Tomcat application backend uses the Kerberos ticket to connect to the Zookeeper node through the Tomcat backend management module and obtain the Hive cluster information in the Hive ZNode. The Hive cluster information includes the node where Hive Server2 is located and the node where HiveMetastore is located. The Tomcat application backend connects to the Hive Metastore through the Tomcat backend management module to obtain the storage path of the Hive library table specified by the user in HDFS and the metadata of the table. The metadata is used for data analysis and reconstruction during the Hive table data backup or recovery process; The backup and recovery management module obtains the request type based on the metadata: if the request type is Hive table data backup, the Kerberos ticket is used to connect to the HDFS Namenode node, and the files under the HDFS storage path corresponding to the Hive table are backed up to the user-specified location in the corresponding format; if the request type is Hive table data recovery, the module connects to the user-specified server to verify the validity of the backup file, and then uses the Kerberos ticket to call the Hive Client API to restore the backup file to the Hive table in the specified format, and update the Hive metadata database information; The Tomcat backend management module updates the relevant backup and recovery data information in the Tomcat application backend data table, and determines whether the backup and recovery task is successfully executed by querying the backend database table through the Tomcat application frontend.

Citation Information

Cited By

  • Data cold and hot switching method for Hive data warehouse

    CN119848051A