HDFS data transfer method and device, storage medium, and electronic device
By automatically configuring Hive metadata, the problem of manually updating Hive metadata after HDFS data transfer is solved, which improves the efficiency and automation of data transfer.
Patent Information
- Application Number
- CN202110685687.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-21
- Publication Date
- 2025-03-21
- Estimated Expiration
- 2041-06-21
AI Technical Summary
When HDFS data is transferred from the source Hadoop cluster to the target Hadoop cluster, the Hive metadata needs to be manually updated, resulting in long and inefficient data transfer time.
By judging whether the name service information of the source Hadoop cluster and the target Hadoop cluster are consistent, if it is inconsistent, the Hive metadata of the HDFS data in the target cluster is automatically configured based on the name service information of the target Hadoop cluster.
It realizes automatic configuration of Hive metadata after HDFS data is transferred to the target Hadoop cluster, avoiding the steps of manual updates and improving the efficiency and automation of data transfer.
Smart Images

Figure CN113434462B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cloud computing, and in particular to a method and device for transferring HDFS data, a storage medium, and an electronic device. Background Art
[0002] In the related technology, Hadoop implements a distributed file system (HDFS, Hadoop Distributed File System), one of whose components is HDFS. HDFS has the characteristics of high fault tolerance and is designed to be deployed on low-cost hardware; it also provides high throughput to access application data and is suitable for applications with very large data sets. Hive is a database service based on HDFS storage.
[0003] In the related art, in the Hive metadata, the data storage path records the complete URL (Uniform Resource Locator) information with the HDFS nameservice (name service), for example: hdfs: / / hadoop-demo / path / to / hive / data / . When backing up or migrating HDFS data, the Hive metadata needs to be backed up and migrated at the same time. If the HDFS nameservice of the new cluster is inconsistent with the old cluster, in order to directly use and query the HDFS data in the new Hadoop cluster, it is necessary to modify the entire URL information and manually rewrite the Hive metadata of the HDFS data in the new cluster one by one, resulting in a long data transfer time.
[0004] With respect to the above-mentioned problems existing in the related technologies, no effective solutions have been found so far. Summary of the invention
[0005] The embodiments of the present invention provide a method and device for transferring HDFS data, a storage medium, and an electronic device.
[0006] According to one aspect of an embodiment of the present application, a method for transferring HDFS data is provided, including: transferring Hadoop distributed file system HDFS data from a source Hadoop cluster to a target Hadoop cluster; determining whether first name service information of the source Hadoop cluster is consistent with second name service information of the target Hadoop cluster; if it is determined that the first name service information of the source Hadoop cluster is inconsistent with the second name service information of the target Hadoop cluster, configuring the HDFS data in the data warehouse Hive metadata of the target Hadoop cluster based on the second name service information.
[0007] Further, configuring the Hive metadata of the data warehouse of the HDFS data in the target Hadoop cluster based on the second name service information includes: querying a specified configuration file in the server node of the target Hadoop cluster, wherein the specified configuration file is used to store Hive configuration parameters of the target Hadoop cluster; and using the second name service information in the specified configuration file to configure a first metadata field of the HDFS data in the target Hadoop cluster, wherein the first metadata field is used to represent the name node IP to which the HDFS data points.
[0008] Furthermore, after transferring the HDFS data from the source Hadoop cluster to the target Hadoop cluster, the method further includes: obtaining a database address of the HDFS data in the source Hadoop cluster; and configuring Hive metadata of the HDFS data in the target Hadoop cluster based on the database address.
[0009] Further, configuring the Hive metadata of the HDFS data in the target Hadoop cluster based on the database address includes: copying the database address to the database of the target Hadoop cluster, and configuring the database address as the Hive metadata of the HDFS data in the target Hadoop cluster, wherein the database address corresponds to the uniform resource locator URL address of the HDFS data in the target Hadoop cluster.
[0010] Furthermore, before transferring the HDFS data from the source Hadoop cluster to the target Hadoop cluster, the method also includes: configuring a first metadata field of the HDFS data in a designated configuration file of a server node of the source Hadoop cluster, and configuring a second metadata field of the HDFS data in a database of the source Hadoop cluster, wherein the Hive metadata of the HDFS data includes the first metadata field and the second metadata field, the first metadata field is used to represent the name node IP pointed to by the HDFS data, and the second metadata field is used to represent the storage path of the HDFS data within the cluster.
[0011] Furthermore, after configuring the data warehouse Hive metadata of the HDFS data in the target Hadoop cluster based on the second name service information, the method also includes: receiving a query request for the HDFS data; obtaining the second name service information, and obtaining a database address of the HDFS data in the target Hadoop cluster; using the second name service information and the database address to splice and generate the Hive metadata of the HDFS data; and locating the HDFS data in the target Hadoop cluster based on the Hive metadata.
[0012] Furthermore, before transferring the HDFS data from the source Hadoop cluster to the target Hadoop cluster, the method further includes one of the following: receiving a backup request for the HDFS data; receiving a migration request for the HDFS data.
[0013] According to another aspect of an embodiment of the present application, a device for transferring HDFS data is also provided, including: a transfer module, used to transfer Hadoop distributed file system HDFS data from a source Hadoop cluster to a target Hadoop cluster; a judgment module, used to judge whether the first name service information of the source Hadoop cluster is consistent with the second name service information of the target Hadoop cluster; a first configuration module, used to configure the HDFS data in the data warehouse Hive metadata of the target Hadoop cluster based on the second name service information if it is judged that the first name service information of the source Hadoop cluster is inconsistent with the second name service information of the target Hadoop cluster.
[0014] Furthermore, the first configuration module includes: a query unit, used to query a specified configuration file in a server node of the target Hadoop cluster, wherein the specified configuration file is used to store Hive configuration parameters of the target Hadoop cluster; a configuration unit, used to configure a first metadata field of the HDFS data in the target Hadoop cluster using the second name service information in the specified configuration file, wherein the first metadata field is used to represent the name node IP pointed to by the HDFS data.
[0015] Furthermore, the device also includes: a transfer module, which is used to obtain the database address of the HDFS data in the source Hadoop cluster after the transfer module transfers the HDFS data from the source Hadoop cluster to the target Hadoop cluster; and a second configuration module, which is used to configure the Hive metadata of the HDFS data in the target Hadoop cluster based on the database address.
[0016] Furthermore, the second configuration module includes: a copying unit, used to copy the database address to the database of the target Hadoop cluster, and configure the database address as Hive metadata of the HDFS data in the target Hadoop cluster, wherein the database address corresponds to the uniform resource locator URL address of the HDFS data in the target Hadoop cluster.
[0017] Furthermore, the device also includes: a third configuration module, which is used to configure a first metadata field of the HDFS data in a designated configuration file of a server node of the source Hadoop cluster and a second metadata field of the HDFS data in a database of the source Hadoop cluster before the transfer module transfers the HDFS data from the source Hadoop cluster to the target Hadoop cluster, wherein the Hive metadata of the HDFS data includes the first metadata field and the second metadata field, the first metadata field is used to represent the name node IP pointed to by the HDFS data, and the second metadata field is used to represent the storage path of the HDFS data within the cluster.
[0018] Furthermore, the device also includes: a receiving module, which is used to receive a query request for the HDFS data after the first configuration module configures the data warehouse Hive metadata of the HDFS data in the target Hadoop cluster based on the second name service information; an acquisition module, which is used to obtain the second name service information and obtain the database address of the HDFS data in the target Hadoop cluster; a generation module, which is used to generate the Hive metadata of the HDFS data by splicing the second name service information and the database address; and a positioning module, which is used to locate the HDFS data in the target Hadoop cluster based on the Hive metadata.
[0019] Furthermore, the device also includes one of the following: a second receiving module, used to receive a backup request for the HDFS data before the transfer module transfers the HDFS data from the source Hadoop cluster to the target Hadoop cluster; a third receiving module, used to receive a migration request for the HDFS data before the transfer module transfers the HDFS data from the source Hadoop cluster to the target Hadoop cluster.
[0020] According to another aspect of an embodiment of the present application, a storage medium is further provided, which includes a stored program, and the above steps are executed when the program is run.
[0021] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the program stored in the memory.
[0022] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps in the above method.
[0023] By means of the present invention, Hadoop distributed file system HDFS data is transferred from a source Hadoop cluster to a target Hadoop cluster, and then it is determined whether the first name service information of the source Hadoop cluster is consistent with the second name service information of the target Hadoop cluster. If it is determined that the first name service information of the source Hadoop cluster is inconsistent with the second name service information of the target Hadoop cluster, the data warehouse Hive metadata of the HDFS data in the target Hadoop cluster is configured based on the second name service information. By determining whether the name service information of the source Hadoop cluster is consistent with the name service information of the target Hadoop cluster, if they are inconsistent, the Hive metadata is configured based on the second name service information, a solution for automatically configuring the Hive metadata after transferring the HDFS data is implemented, the technical problem of the related art that the Hive metadata needs to be manually configured after the data is transferred is solved, and the configuration efficiency of the Hive metadata is improved. After the HDFS data is transferred to the target Hadoop cluster, it does not need to be manually updated and can be directly used and queried in the target Hadoop cluster. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0025] Figure 1 is a hardware structure block diagram of a server according to an embodiment of the present invention;
[0026] Figure 2 is a flow chart of a method for transferring HDFS data according to an embodiment of the present invention;
[0027] Figure 3 is a transfer flow chart of an optional embodiment of the present invention;
[0028] Figure 4 is a structural block diagram of a device for transferring HDFS data according to an embodiment of the present invention;
[0029] Figure 5 It is a structural block diagram of an electronic device implementing an embodiment of the present invention. DETAILED DESCRIPTION
[0030] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.
[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0032] Example 1
[0033] The method embodiment provided in the first embodiment of the present application can be executed in a server, a computer, or a similar computing device. Taking running on a server as an example, Figure 1 FIG. 1 is a hardware structure diagram of a server according to an embodiment of the present invention. Figure 1 As shown, the server may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the server may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.
[0034] The memory 104 can be used to store server programs, for example, software programs and modules of application software, such as a server program corresponding to a method for transferring HDFS data in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the server program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0035] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0036] In this embodiment, a method for transferring HDFS data is provided. Figure 2 is a flow chart of a method for transferring HDFS data according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:
[0037] Step S202, transferring Hadoop distributed file system HDFS data from the source Hadoop cluster to the target Hadoop cluster;
[0038] The HDFS data and Hive metadata of this embodiment are collectively referred to as Hive data, which is a component element of the Hive service in the Hadoop cluster. The Hive metadata is used to record the library table information in Hive and the storage path of the data file in HDFS in a relational database (such as MySQL). Each HDFS data corresponds to a Hive metadata.
[0039] Step S204, determining whether the first name service information of the source Hadoop cluster is consistent with the second name service information of the target Hadoop cluster;
[0040] The name service of this embodiment is a kind of configuration information in the Hadoop cluster, which is a URL alias when accessing HDFS. In actual configuration, it will point to the NameNode IP. When operating HDFS, there is no need to operate the IP, but use the nameservice instead.
[0041] Step S206: If it is determined that the first name service information of the source Hadoop cluster is inconsistent with the second name service information of the target Hadoop cluster, configure the Hive metadata of the data warehouse of the HDFS data in the target Hadoop cluster based on the second name service information.
[0042] On the other hand, if it is determined that the first name service information of the source Hadoop cluster is consistent with the second name service information of the target Hadoop cluster, the Hive metadata of the HDFS data in the target Hadoop cluster is consistent with that in the source Hadoop cluster and can be used directly.
[0043] Through the above steps, the Hadoop distributed file system HDFS data is transferred from the source Hadoop cluster to the target Hadoop cluster, and then it is determined whether the first name service information of the source Hadoop cluster is consistent with the second name service information of the target Hadoop cluster. If it is determined that the first name service information of the source Hadoop cluster is inconsistent with the second name service information of the target Hadoop cluster, the data warehouse Hive metadata of the HDFS data in the target Hadoop cluster is configured based on the second name service information. By determining whether the name service information of the source Hadoop cluster is consistent with that of the target Hadoop cluster, if it is inconsistent, the Hive metadata is configured based on the second name service information, a solution for automatically configuring the Hive metadata after transferring the HDFS data is implemented, which solves the technical problem of the related technology that the Hive metadata needs to be manually configured after the data is transferred, and improves the configuration efficiency of the Hive metadata. After the HDFS data is transferred to the target Hadoop cluster, it does not need to be manually updated and can be directly used and queried in the target Hadoop cluster.
[0044] In this embodiment, configuring the Hive metadata of the data warehouse of the HDFS data in the target Hadoop cluster based on the second name service information includes:
[0045] S11, querying a specified configuration file in a server node of a target Hadoop cluster, wherein the specified configuration file is used to store Hive configuration parameters of the target Hadoop cluster;
[0046] Optionally, the specified configuration file is a hive-site.xml file in a server node (service node), and of course, it can also be other custom files in a network node of the target Hadoop cluster.
[0047] S12, using the second name service information in the specified configuration file to configure a first metadata field of the HDFS data in the target Hadoop cluster, wherein the first metadata field is used to represent the name node IP to which the HDFS data points.
[0048] In an example, the first name service information is hadoop-demo1, and the second name service information is hadoop-demo2. If the designated configuration file of the target Hadoop cluster has configured the second name service information, hadoop-demo2 can be directly used as part of the Hive metadata of the newly transferred HDFS data, that is, the first metadata field; if the designated configuration file of the target Hadoop cluster has not configured the second name service information, hadoop-demo2 is newly added to the designated configuration file, and then hadoop-demo2 can be used as part of the Hive metadata of the newly transferred HDFS data.
[0049] In another aspect of this embodiment, after transferring the HDFS data from the source Hadoop cluster to the target Hadoop cluster, the method further includes:
[0050] S21, obtaining the database address of the HDFS data in the source Hadoop cluster;
[0051] Taking the relational database MySQL as an example, the database address of HDFS data in the source Hadoop cluster is path / to / hive / data / , which is stored in MySQL. If other data storage methods are used, the database address can also be stored in other locations of the source Hadoop cluster. For example, if the derby memory database is used to save hive metadata, a derby file and a metastore_db directory are generated in the current directory when the derby storage method is used. The database address will be stored in the metastore_db directory.
[0052] S22, configure Hive metadata of HDFS data in the target Hadoop cluster based on the database address.
[0053] In one example, if the source Hadoop cluster and the target Hadoop cluster both use the same type of database, such as MySQL, since the database itself is universal, the database address can be directly used in the target Hadoop cluster. If the database A used by the source Hadoop cluster and the database B used by the target Hadoop cluster are different, the database address needs to be converted to the database address of database B. The database address is another part of the Hive metadata of the HDFS data, namely the second metadata field. The second metadata field and the first metadata field constitute the complete Hive metadata.
[0054] In one implementation of this embodiment, configuring the data warehouse Hive metadata of HDFS data in the target Hadoop cluster based on the database address includes: copying the database address to the database of the target Hadoop cluster, and configuring the database address as the Hive metadata of the HDFS data in the target Hadoop cluster, wherein the database address corresponds to the uniform resource locator URL address of the HDFS data in the target Hadoop cluster.
[0055] In one implementation of the present embodiment, before transferring HDFS data from a source Hadoop cluster to a target Hadoop cluster, the process further includes: configuring a first metadata field of the HDFS data in a designated configuration file of a server node of the source Hadoop cluster, and configuring a second metadata field of the HDFS data in a database of the source Hadoop cluster, wherein the Hive metadata of the HDFS data includes a first metadata field and a second metadata field, the first metadata field being used to represent the name node IP to which the HDFS data points, and the second metadata field being used to represent the storage path of the HDFS data within the cluster.
[0056] In some application scenarios of this embodiment, the target Hadoop cluster uses Hive metadata to query the HDFS data. After configuring the data warehouse Hive metadata of the HDFS data in the target Hadoop cluster based on the second name service information, the method further includes: receiving a query request for HDFS data; obtaining the second name service information, and obtaining the database address of the HDFS data in the target Hadoop cluster; using the second name service information and the database address to generate the Hive metadata of the HDFS data; and locating the HDFS data in the target Hadoop cluster based on the Hive metadata.
[0057] In an application scenario, the user sends a query request to the cluster (including the target Hadoop cluster) through the client to query the target HDFS data, and searches the databases in each cluster through the file identifier or other keywords of the target HDFS data. The target HDFS data is found in the path path / to / hive / data / of the target Hadoop cluster, and its nameservice (hadoop-demo2) is obtained in the specified configuration file of the target Hadoop cluster. Through composition, the complete Hive metadata - hdfs: / / hadoop-demo2 / path / to / hive / data is obtained, and it is fed back to the client. The client uses the Hive metadata to locate the HDFS data and choose to perform further operations on the HDFS data.
[0058] The solution of this embodiment can be applied to data transfer scenarios such as backup or migration of HDFS data. Before transferring HDFS data from a source Hadoop cluster to a target Hadoop cluster, it also includes: receiving a backup request for HDFS data; or receiving a migration request for HDFS data.
[0059] Figure 3 FIG. 1 is a transfer flow chart of an optional embodiment of the present invention. Figure 3 As shown, it can be understood that in some Hive data backup / migration processes, the process of backing up / migrating Hive data may include an HDFS data transfer (backup / migration) process and a Hive metadata transfer process, which may include the following steps:
[0060] S31, back up / migrate HDFS data to the target Hadoop cluster;
[0061] S32, back up / migrate MySQL data (i.e. Hive metadata) to the target MySQL;
[0062] S33, verify whether the nameservice information is consistent with the target Hadoop cluster;
[0063] S34, if they are consistent, determine that the migration is complete; if they are inconsistent, update the nameservice information of the target Hadoop cluster to be consistent with the target cluster.
[0064] An improvement in an optional embodiment of the invention is that, in step S33, the nameservice in the metadata is configured separately and configured separately in the Hive configuration file (hive-site.xml), so that the nameservice does not need to be fixed in the metadata. When performing data backup / migration, since the nameservice in the Hive configuration file in the target Hadoop cluster is originally correctly configured, there is no need to make additional adjustments, and it is directly configured as a part of the Hive metadata of the HDFS data in the target cluster, thereby eliminating steps S33 and S34, that is, eliminating the step of verifying the nameservice information.
[0065] By adopting the solution of this embodiment, nameservice is separated from metadata and defined in hive-site.xml as a separate Hive configuration. It does not need to be recorded in metadata. When migrating or backing up Hive data, only hive-site.xml needs to be adjusted in the target cluster, and the content in MySQL does not need to be updated. At the same time, the time for data backup / migration is saved. After the HDFS data + Hive metadata is migrated, it can be directly used and queried in the target Hadoop cluster without manually updating the nameservice in the Hive metadata one by one.
[0066] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0067] Example 2
[0068] In this embodiment, a device for transferring HDFS data is also provided to implement the above-mentioned embodiments and preferred implementation modes, which have been described and will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0069] Figure 4is a structural block diagram of a device for transferring HDFS data according to an embodiment of the present invention. Figure 4 As shown, the device includes: a transfer module 40, a judgment module 42, and a first configuration module 44, wherein:
[0070] A transfer module 40, used to transfer Hadoop distributed file system HDFS data from a source Hadoop cluster to a target Hadoop cluster;
[0071] A determination module 42, configured to determine whether the first name service information of the source Hadoop cluster is consistent with the second name service information of the target Hadoop cluster;
[0072] The first configuration module 44 is configured to configure the Hive metadata of the data warehouse of the HDFS data in the target Hadoop cluster based on the second name service information if it is determined that the first name service information of the source Hadoop cluster is inconsistent with the second name service information of the target Hadoop cluster.
[0073] Optionally, the first configuration module includes: a query unit, used to query a specified configuration file in the server node of the target Hadoop cluster, wherein the specified configuration file is used to store Hive configuration parameters of the target Hadoop cluster; a configuration unit, used to configure a first metadata field of the HDFS data in the target Hadoop cluster using the second name service information in the specified configuration file, wherein the first metadata field is used to represent the name node IP pointed to by the HDFS data.
[0074] Optionally, the device also includes: a transfer module, used to obtain the database address of the HDFS data in the source Hadoop cluster after the transfer module transfers the HDFS data from the source Hadoop cluster to the target Hadoop cluster; and a second configuration module, used to configure the Hive metadata of the HDFS data in the target Hadoop cluster based on the database address.
[0075] Optionally, the second configuration module includes: a copying unit, used to copy the database address to the database of the target Hadoop cluster, and configure the database address as Hive metadata of the HDFS data in the target Hadoop cluster, wherein the database address corresponds to the uniform resource locator URL address of the HDFS data in the target Hadoop cluster.
[0076] Optionally, the device also includes: a third configuration module, used to configure a first metadata field of the HDFS data in a designated configuration file of a server node of the source Hadoop cluster and a second metadata field of the HDFS data in a database of the source Hadoop cluster before the transfer module transfers the HDFS data from the source Hadoop cluster to the target Hadoop cluster, wherein the Hive metadata of the HDFS data includes the first metadata field and the second metadata field, the first metadata field is used to represent the name node IP pointed to by the HDFS data, and the second metadata field is used to represent the storage path of the HDFS data within the cluster.
[0077] Optionally, the device also includes: a receiving module, used to receive a query request for the HDFS data after the first configuration module configures the data warehouse Hive metadata of the HDFS data in the target Hadoop cluster based on the second name service information; an acquisition module, used to obtain the second name service information and obtain the database address of the HDFS data in the target Hadoop cluster; a generation module, used to generate the Hive metadata of the HDFS data by splicing the second name service information and the database address; and a positioning module, used to locate the HDFS data in the target Hadoop cluster based on the Hive metadata.
[0078] Optionally, the device also includes one of the following: a second receiving module, used to receive a backup request for the HDFS data before the transfer module transfers the HDFS data from the source Hadoop cluster to the target Hadoop cluster; a third receiving module, used to receive a migration request for the HDFS data before the transfer module transfers the HDFS data from the source Hadoop cluster to the target Hadoop cluster.
[0079] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0080] Example 3
[0081] The embodiment of the present application also provides an electronic device, Figure 5 is a structural diagram of an electronic device according to an embodiment of the present invention, such as Figure 5As shown, it includes a processor 51, a communication interface 52, a memory 53 and a communication bus 54, wherein the processor 51, the communication interface 52 and the memory 53 communicate with each other through the communication bus 54, and the memory 53 is used to store computer programs; when the processor 51 is used to execute the program stored in the memory 53, the following steps are implemented: transferring Hadoop distributed file system HDFS data from a source Hadoop cluster to a target Hadoop cluster; determining whether the first name service information of the source Hadoop cluster is consistent with the second name service information of the target Hadoop cluster; if it is determined that the first name service information of the source Hadoop cluster is inconsistent with the second name service information of the target Hadoop cluster, configuring the HDFS data in the data warehouse Hive metadata of the target Hadoop cluster based on the second name service information.
[0082] Further, configuring the Hive metadata of the data warehouse of the HDFS data in the target Hadoop cluster based on the second name service information includes: querying a specified configuration file in the server node of the target Hadoop cluster, wherein the specified configuration file is used to store Hive configuration parameters of the target Hadoop cluster; and using the second name service information in the specified configuration file to configure a first metadata field of the HDFS data in the target Hadoop cluster, wherein the first metadata field is used to represent the name node IP to which the HDFS data points.
[0083] Furthermore, after transferring the HDFS data from the source Hadoop cluster to the target Hadoop cluster, the method further includes: obtaining a database address of the HDFS data in the source Hadoop cluster; and configuring Hive metadata of the HDFS data in the target Hadoop cluster based on the database address.
[0084] Further, configuring the Hive metadata of the HDFS data in the target Hadoop cluster based on the database address includes: copying the database address to the database of the target Hadoop cluster, and configuring the database address as the Hive metadata of the HDFS data in the target Hadoop cluster, wherein the database address corresponds to the uniform resource locator URL address of the HDFS data in the target Hadoop cluster.
[0085] Furthermore, before transferring the HDFS data from the source Hadoop cluster to the target Hadoop cluster, the method also includes: configuring a first metadata field of the HDFS data in a designated configuration file of a server node of the source Hadoop cluster, and configuring a second metadata field of the HDFS data in a database of the source Hadoop cluster, wherein the Hive metadata of the HDFS data includes the first metadata field and the second metadata field, the first metadata field is used to represent the name node IP pointed to by the HDFS data, and the second metadata field is used to represent the storage path of the HDFS data within the cluster.
[0086] Furthermore, after configuring the data warehouse Hive metadata of the HDFS data in the target Hadoop cluster based on the second name service information, the method also includes: receiving a query request for the HDFS data; obtaining the second name service information, and obtaining a database address of the HDFS data in the target Hadoop cluster; using the second name service information and the database address to splice and generate the Hive metadata of the HDFS data; and locating the HDFS data in the target Hadoop cluster based on the Hive metadata.
[0087] Furthermore, before transferring the HDFS data from the source Hadoop cluster to the target Hadoop cluster, the method further includes one of the following: receiving a backup request for the HDFS data; receiving a migration request for the HDFS data.
[0088] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0089] The communication interface is used for communication between the above terminal and other devices.
[0090] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0091] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0092] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is executed on a computer, the computer executes the HDFS data transfer method described in any of the above embodiments.
[0093] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the HDFS data transfer method described in any one of the above embodiments.
[0094] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.
[0095] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
[0096] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.
Claims
1. A method for transferring HDFS data, characterized in that: include: Transfer Hadoop Distributed File System HDFS data from the source Hadoop cluster to the target Hadoop cluster; Determine whether the first name service information of the source Hadoop cluster is consistent with the second name service information of the target Hadoop cluster; If it is determined that the first name service information of the source Hadoop cluster is inconsistent with the second name service information of the target Hadoop cluster, configuring the Hive metadata of the data warehouse of the HDFS data in the target Hadoop cluster based on the second name service information; Configuring the Hive metadata of the data warehouse of the HDFS data in the target Hadoop cluster based on the second name service information includes: querying a specified configuration file in the server node of the target Hadoop cluster, wherein the specified configuration file is used to store Hive configuration parameters of the target Hadoop cluster; and configuring a first metadata field of the HDFS data in the target Hadoop cluster using the second name service information in the specified configuration file, wherein the first metadata field is used to represent the name node IP to which the HDFS data points.
2. The method according to claim 1, characterized in that: After transferring the HDFS data from the source Hadoop cluster to the target Hadoop cluster, the method further includes: Obtaining a database address of the HDFS data in the source Hadoop cluster; Hive metadata of the HDFS data is configured in the target Hadoop cluster based on the database address.
3. The method according to claim 2, characterized in that Configuring the Hive metadata of the HDFS data in the target Hadoop cluster based on the database address includes: The database address is copied to a database of the target Hadoop cluster, and the database address is configured as Hive metadata of the HDFS data in the target Hadoop cluster, wherein the database address corresponds to a uniform resource locator URL address of the HDFS data in the target Hadoop cluster.
4. The method according to claim 1, characterized in that: Before transferring the HDFS data from the source Hadoop cluster to the target Hadoop cluster, the method further includes: A first metadata field of the HDFS data is configured in a designated configuration file of a server node of the source Hadoop cluster, and a second metadata field of the HDFS data is configured in a database of the source Hadoop cluster, wherein the Hive metadata of the HDFS data includes the first metadata field and the second metadata field, the first metadata field is used to characterize the name node IP pointed to by the HDFS data, and the second metadata field is used to characterize the storage path of the HDFS data in the cluster.
5. The method according to claim 1, characterized in that After configuring the data warehouse Hive metadata of the HDFS data in the target Hadoop cluster based on the second name service information, the method further includes: Receive a query request for the HDFS data; Obtain the second name service information, and obtain the database address of the HDFS data in the target Hadoop cluster; Generate Hive metadata of the HDFS data by using the second name service information and the database address; The HDFS data is located in a target Hadoop cluster based on the Hive metadata.
6. The method according to claim 1, characterized in that Before transferring the HDFS data from the source Hadoop cluster to the target Hadoop cluster, the method further includes one of the following: Receiving a backup request for the HDFS data; A migration request for the HDFS data is received.
7. A device for transferring HDFS data, characterized in that: include: A transfer module is used to transfer Hadoop distributed file system HDFS data from a source Hadoop cluster to a target Hadoop cluster; A determination module, configured to determine whether the first name service information of the source Hadoop cluster is consistent with the second name service information of the target Hadoop cluster; A first configuration module is configured to configure the Hive metadata of the data warehouse of the HDFS data in the target Hadoop cluster based on the second name service information if it is determined that the first name service information of the source Hadoop cluster is inconsistent with the second name service information of the target Hadoop cluster; The first configuration module includes: a query unit, used to query a specified configuration file in a server node of the target Hadoop cluster, wherein the specified configuration file is used to store Hive configuration parameters of the target Hadoop cluster; a configuration unit, used to configure a first metadata field of the HDFS data in the target Hadoop cluster using the second name service information in the specified configuration file, wherein the first metadata field is used to represent the name node IP pointed to by the HDFS data.
8. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 6 when executed.
9. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface, and the memory communicate with each other via a communication bus; wherein: Memory, used to store computer programs; A processor, configured to execute the method according to any one of claims 1 to 6 by running a program stored in a memory.
