HDFS data transfer method and device, storage medium, and electronic device
By automatically configuring Hive metadata, the problem of manually updating Hive metadata after HDFS data transfer is solved, which improves the efficiency and automation of data transfer.
CN113434462BActive Publication Date: 2025-03-21BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD
0 Cites 0 Cited by
Patent Information
- Application Number
- CN202110685687.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-21
- Publication Date
- 2025-03-21
- Estimated Expiration
- 2041-06-21
AI Technical Summary
Technical Problem
When HDFS data is transferred from the source Hadoop cluster to the target Hadoop cluster, the Hive metadata needs to be manually updated, resulting in long and inefficient data transfer time.
Method used
By judging whether the name service information of the source Hadoop cluster and the target Hadoop cluster are consistent, if it is inconsistent, the Hive metadata of the HDFS data in the target cluster is automatically configured based on the name service information of the target Hadoop cluster.
Benefits of technology
It realizes automatic configuration of Hive metadata after HDFS data is transferred to the target Hadoop cluster, avoiding the steps of manual updates and improving the efficiency and automation of data transfer.
✦ Generated by Eureka AI based on patent content.
Abstract
The present invention discloses a method and device for transferring HDFS data, a storage medium, and an electronic device, and belongs to the field of cloud computing. The method includes: transferring Hadoop distributed file system HDFS data from a source Hadoop cluster to a target Hadoop cluster; judging whether the first name service information of the source Hadoop cluster is consistent with the second name service information of the target Hadoop cluster; if it is judged that the first name service information of the source Hadoop cluster is inconsistent with the second name service information of the target Hadoop cluster, configuring the HDFS data in the data warehouse Hi ve metadata of the target Hadoop cluster based on the second name service information. The present invention solves the technical problem that the Hi ve metadata needs to be manually configured after data transfer in the related technology, improves the configuration efficiency of the Hi ve metadata, and after the HDFS data is transferred to the target Hadoop cluster, it does not need to be manually updated, and can be directly used and queried in the target Hadoop cluster.
Need to check novelty before this filing date? Find Prior Art