A method and apparatus for smoothly migrating data of Hadoop to object storage

Abstract

The application discloses a method and device for smoothly migrating data of Hadoop to object storage, wherein the method comprises the following steps: suspending a Hadoop write operation, backing up metadata of a master NameNode, recording a current maximum offset of an edit log of a JournalNode of HDFS, offlineing one copy of each Block block in the backed-up NameNode metadata, and resuming the Hadoop write operation; analyzing the backed-up NameNode metadata, establishing a mapping relationship between a DataNode and the offline Block block copy, decomposing a data synchronization task, synchronizing data and limiting a data synchronization rate; taking the recorded current maximum offset of the edit log of the JournalNode of HDFS as a starting point, synchronizing operation information of data change recorded by the edit log of the JournalNode of HDFS, incrementally synchronizing data based on the operation information of the data change, and smoothly migrating. The method and device can smoothly migrate data of Hadoop to object storage, can perform distributed data synchronization, do not affect the operation efficiency of normal online services, and support hot migration.

Description

Technical Field

[0001] This invention relates to the field of object storage architecture, and in particular to a method and apparatus for smoothly migrating data from Hadoop to object storage. Background Technology

[0002] 1. Why migrate data from Hadoop to object storage?

[0003] Traditional big data platforms are generally based on Hadoop. However, with the development of modern data technology stacks, problems with Hadoop have become increasingly apparent, mainly including:

[0004] (1) Storage management complexity: Hadoop Distributed File System (HDFS) needs to manage block storage and requires periodic operations such as data block replication and sharding, which increases the complexity of storage management.

[0005] (2) High hardware cost: Hadoop clusters typically require a large number of physical servers as well as additional network and storage devices, which results in relatively high hardware costs.

[0006] (3) Scalability challenges: Although Hadoop can scale horizontally, adding new nodes is not always simple. Configuring, tuning, and maintaining new nodes all require technical knowledge, which can increase the complexity of deployment and maintenance.

[0007] (4) Resource waste: In Hadoop, each node typically stores multiple copies of data to achieve fault tolerance. This can lead to a waste of storage resources, especially for less important data or data that does not require high availability.

[0008] (5) Data access speed: Traditional Hadoop may have performance issues when processing small files because it needs to manage the metadata of a large number of small files, which may result in low data access speed.

[0009] (6) Data storage efficiency: Hadoop typically uses block storage, which may result in low data storage efficiency, especially for sparse data and small files.

[0010] In contrast, object storage (OSS) can alleviate these problems to some extent. Object storage typically uses a distributed, flat storage architecture, eliminating the need for complex management and replication like HDFS. Object storage also offers faster data access speeds, is suitable for storing large numbers of small and large files, and is applicable to a variety of use cases, including backup, archiving, image and video storage, data lakes, and content distribution. It can be used to store different types of data, not just big data processing. Object storage facilitates global data distribution, making data migration between different geographical locations easier and helping to meet compliance and data locality requirements. Furthermore, object storage generally offers better scalability and flexibility and is more common in cloud environments, simplifying resource configuration and management.

[0011] 2. Challenges of migrating data from Hadoop to object storage

[0012] Given the aforementioned problems with Hadoop and the advantages of object storage, an increasing number of enterprises are upgrading their big data platforms from Hadoop to object storage. However, the smooth migration of data from Hadoop to object storage presents the following challenges:

[0013] (1) Low synchronization efficiency: Hadoop stores a very large amount of data, even exceeding the petabyte level. Migrating such a large amount of data to object storage results in long synchronization time and low efficiency.

[0014] (2) Affects the operational efficiency of normal online business: Massive data synchronization will lead to extremely high disk I / O and network I / O usage, and may even affect the operational efficiency of normal online business.

[0015] (3) Difficult to support hot migration: If the service is not stopped during data migration, the big data cluster will be constantly uploading, deleting and moving data during the migration period, making it difficult to guarantee the consistency of data before and after the migration. In order to ensure the consistency of data in Hadoop and object storage, the Hadoop service usually needs to be stopped during the migration. The longer the service is stopped, the greater the impact on online business. Summary of the Invention

[0016] To address the aforementioned problems in the existing technology, this invention provides a method and apparatus for smoothly migrating Hadoop data to object storage. This method enables distributed data synchronization, does not affect the operational efficiency of normal online business, and supports hot migration.

[0017] To achieve the above objectives, the present invention adopts the following technical solution:

[0018] In one embodiment of the present invention, a method for smoothly migrating data from Hadoop to object storage is proposed, the method comprising:

[0019] Create a copy of NameNode metadata: Pause Hadoop write operations, back up metadata from the primary NameNode, record the current maximum offset of the HDFS JournalNode's edit log, take any block copy of the backed-up NameNode metadata offline, and resume Hadoop write operations.

[0020] Existing data synchronization: Parse the backed-up NameNode metadata, establish the mapping relationship between DataNote nodes and offline Block replicas, decompose the data synchronization task, synchronize data and limit the data synchronization rate;

[0021] Incremental data synchronization: Starting from the current maximum offset of the edit log of the HDFS JournalNode, the operation information of data change recorded in the edit log of the HDFS JournalNode is synchronized incrementally based on the operation information of data change, so as to achieve smooth migration.

[0022] Furthermore, when the progress of data synchronization to object storage is consistent with the progress of data writing to Hadoop, the data entry program, cleaning program, and analysis program are connected to object storage.

[0023] Furthermore, the data synchronization task is broken down into the following parts:

[0024] If the files that need to be synchronized by the DataNode node and the offline Block replicas are one-to-one, the main service will distribute the data synchronization task to the DataNode node where the offline Block replicas are located.

[0025] If the files to be synchronized by a DataNode and the offline block replicas are in a one-to-many relationship, the main service will distribute the data synchronization task to the DataNode where the largest offline block replica is located. If there are two offline block replicas of the same size, the DataNode where the offline block replica is located will be randomly selected.

[0026] Furthermore, data synchronization includes:

[0027] If the file that the current DataNode needs to synchronize corresponds to only one offline Block replica, then the slave service deployed on the DataNode reads the data of the offline Block replica from the disk of the current DataNode and sends it to object storage.

[0028] If the file that the current DataNode needs to synchronize corresponds to two or more offline Block replicas, the slave service deployed on the DataNode will read the data of the offline Block replicas from the disk of the current DataNode, pull the data of the offline Block replicas from other DataNodes, concatenate the data of these offline Block replicas, and send them to object storage.

[0029] Furthermore, limiting the rate of data synchronization includes:

[0030] On each HDFS DataNode, a data synchronization queue is initialized with a fixed capacity; the elements in the data synchronization queue are used to trigger a data synchronization request.

[0031] Elements in the data synchronization queue are generated at a fixed rate, and the generated elements are placed into the data synchronization queue.

[0032] The data synchronization queue has a capacity limit. If the data synchronization queue is full and no new elements are generated, the extra data synchronization requests will not be able to obtain the elements, thus limiting the data synchronization rate.

[0033] In one embodiment of the present invention, an apparatus for smoothly migrating data from Hadoop to object storage is also provided, the apparatus comprising:

[0034] The NameNode metadata copy module is used to pause Hadoop write operations, back up metadata from the primary NameNode, record the current maximum offset of the HDFS JournalNode's edit log, take down any block replica in the backed-up NameNode metadata, and resume Hadoop write operations.

[0035] The existing data synchronization module is used to parse the backed-up NameNode metadata, establish the mapping relationship between DataNote nodes and offline Block replicas, decompose data synchronization tasks, synchronize data, and limit the data synchronization rate.

[0036] The incremental data synchronization module is used to synchronize the data change operation information recorded in the HDFS JournalNode's edit log, starting from the current maximum offset of the recorded HDFS JournalNode's edit log. Based on the data change operation information, the module incrementally synchronizes data for smooth migration.

[0037] Furthermore, when the progress of data synchronization to object storage is consistent with the progress of data writing to Hadoop, the data entry program, cleaning program, and analysis program are connected to object storage.

[0038] Furthermore, the data synchronization task is broken down into the following parts:

[0039] If the files that need to be synchronized by the DataNode node and the offline Block replicas are one-to-one, the main service will send the data synchronization task to the DataNode node where the offline Block replicas are located.

[0040] If the files to be synchronized by a DataNode and the offline block replicas are in a one-to-many relationship, the main service will distribute the data synchronization task to the DataNode where the largest offline block replica is located. If there are two offline block replicas of the same size, the DataNode where the offline block replica is located will be randomly selected.

[0041] Furthermore, data synchronization includes:

[0042] If the file that the current DataNode needs to synchronize corresponds to only one offline Block replica, then the slave service deployed on the DataNode reads the data of the offline Block replica from the disk of the current DataNode and sends it to object storage.

[0043] If the file that the current DataNode needs to synchronize corresponds to two or more offline Block replicas, the slave service deployed on the DataNode will read the data of the offline Block replicas from the disk of the current DataNode, pull the data of the offline Block replicas from other DataNodes, concatenate the data of these offline Block replicas, and send them to object storage.

[0044] Furthermore, limiting the rate of data synchronization includes:

[0045] On each HDFS DataNode, a data synchronization queue is initialized with a fixed capacity; the elements in the data synchronization queue are used to trigger a data synchronization request.

[0046] Elements in the data synchronization queue are generated at a fixed rate, and the generated elements are placed into the data synchronization queue.

[0047] The data synchronization queue has a capacity limit. If the data synchronization queue is full and no new elements are generated, the extra data synchronization requests will not be able to obtain the elements, thus limiting the data synchronization rate.

[0048] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned method for smoothly migrating Hadoop data to object storage.

[0049] In one embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that performs a method for smoothly migrating data from Hadoop to object storage.

[0050] Beneficial effects:

[0051] 1. High synchronization efficiency: Data synchronization is performed separately on each DataNode in Hadoop, resulting in high efficiency.

[0052] 2. Does not affect the operational efficiency of normal online business: Massive data synchronization tasks are distributed to each DataNode node, and the data synchronization operation rate can be limited.

[0053] 3. Supports hot migration: During data synchronization, write operations on the big data cluster are only paused briefly during the NameNode metadata copy. After the NameNode metadata copy is complete, the big data cluster continuously performs data upload, deletion, and movement operations, all of which can be synchronized to object storage.

[0054] 4. Smooth Migration: During data synchronization to object storage, data ingestion, cleaning, and analysis processes can be pre-connected to object storage. Since the data synchronization progress to object storage is synchronized with the data writing progress to Hadoop, all processes, including ingestion, cleaning, and analysis, can be seamlessly integrated into object storage, achieving a smooth migration. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the method for smoothly migrating data from Hadoop to object storage according to the present invention;

[0056] Figure 2 This is an architecture diagram of existing data synchronization according to an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the device structure for smoothly migrating data from Hadoop to object storage according to the present invention;

[0058] Figure 4 This is a schematic diagram of the computer device structure of the present invention. Detailed Implementation

[0059] The principles and spirit of the present invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0060] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0061] According to an embodiment of the present invention, a method and apparatus for smoothly migrating Hadoop data to object storage are proposed. The data synchronization from Hadoop to object storage is divided into two steps: (1) synchronizing existing data; (2) synchronizing incremental data after the existing data synchronization is completed. Before synchronizing existing data, write operations on the Hadoop big data cluster are paused, NameNode metadata is copied, and one copy of each block is taken offline. Since the offline block copy no longer undergoes any modification or deletion operations, this block copy is used for existing data synchronization. After copying the NameNode metadata is completed, write operations on the Hadoop big data cluster are resumed. The existing data synchronization operation is distributed to the DataNode nodes of the big data cluster for distributed execution. To avoid impacting online services, the synchronization speed is limited. After copying the NameNode metadata is complete, the Hadoop big data cluster can perform write operations, generating data additions, deletions, modifications, and other changes. These changes are recorded in the Edit Log of the HDFS JournalNode. The JournalNode's Edit Log is periodically synchronized to a message queue (Kafka) or file system. The incremental data synchronization program consumes data from Kafka or the file system, retrieving a batch of change operation information each time, merging the information to form the synchronized data, and then performing incremental data synchronization operations based on this content. This ensures hot data migration while the Hadoop big data cluster remains capable of normal read and write operations. During data synchronization to object storage, data ingestion, cleaning, and analysis programs can be pre-connected to object storage. Once the data synchronization progress to object storage is consistent with the data writing progress to Hadoop, all ingestion, cleaning, and analysis programs can be connected to object storage for a smooth migration.

[0062] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0063] Figure 1 This is a schematic diagram of the method flow for smoothly migrating data from Hadoop to object storage according to the present invention. Figure 1 As shown, the method includes:

[0064] Create a copy of NameNode metadata: Pause Hadoop write operations, back up metadata from the primary NameNode, record the current maximum offset of the HDFS JournalNode's edit log, take any block copy of the backed-up NameNode metadata offline, and resume Hadoop write operations.

[0065] In HDFS, a key component is called the NameNode, which is responsible for managing the file system's metadata (such as information about files and directories). To ensure data recovery even if the NameNode fails, HDFS uses JournalNodes to record all file system operations.

[0066] When write operations are performed on HDFS, such as creating a new file or modifying an existing file, these operations are first logged to the JournalNode. Upon receiving these operations, the JournalNode stores them in a persistent log file and ensures they are replicated to other JournalNodes for redundancy.

[0067] Existing data synchronization: Parse the backed-up NameNode metadata, establish the mapping relationship between DataNote nodes and offline Block replicas, decompose the data synchronization task, synchronize data and limit the data synchronization rate, and the existing data synchronization is completed.

[0068] Incremental data synchronization: Starting from the maximum offset of the current edit log of the HDFS JournalNode, the operation information of data change recorded in the edit log of the HDFS JournalNode is synchronized to the message queue Kafka or file system, etc., and the data is incrementally synchronized based on the operation information of data change for smooth migration.

[0069] The maximum offset allows you to locate the entry in the edit log. After the NameNode metadata copy is complete, the NameNode metadata is recorded at the end of the edit log. Once the full backup is complete, data is read from this recorded position, meaning you read data entries for addition, deletion, and modification.

[0070] Since data is continuously written to Hadoop, and Hadoop data needs to be synchronized to object storage, the data entry, cleaning, and analysis programs can be connected to object storage when the data synchronization progress to object storage is consistent with the data writing progress to Hadoop—that is, when the data written to Hadoop can be synchronized to object storage in a relatively timely manner without being blocked.

[0071] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0072] To provide a clearer explanation of the method for smoothly migrating data from Hadoop to object storage, a specific embodiment will be used for illustration below. However, it is worth noting that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation of the present invention.

[0073] 1. Create a copy of the NameNode metadata

[0074] To ensure high availability and data consistency in Hadoop, write operations on the Hadoop big data cluster are paused during the copying of NameNode metadata, but read operations remain available. Since this operation only involves copying NameNode metadata and is relatively fast, the pause time for write operations is very short, resulting in minimal impact on business operations.

[0075] The specific process is as follows:

[0076] (1) Enable safe mode, stop writing data, but allow reading data.

[0077] (2) Backup metadata from the primary NameNode.

[0078] (3) Record the current maximum offset of the JournalNode's Edit Log.

[0079] (4) Take replica data offline. Parse the backed-up NameNode metadata, and for each block, take one replica offline as the offline replica. Mark the offline block replica as offline in the backed-up NameNode metadata. The offline block replica is no longer used for data querying, modification, deletion, or other operations.

[0080] (5) Exit safe mode and resume writing data.

[0081] After creating the NameNode metadata copy, the following two processes will then proceed:

[0082] Process 1: Copy the existing data to object storage based on the NameNode metadata.

[0083] Step 2: Synchronize data change operation information from the JournalNode, extracting key operations such as adding files, deleting files, adding blocks, and deleting blocks as a file system change operation log. Starting from the maximum offset of the JournalNode's Edit Log recorded during the NameNode metadata copying process, synchronize data change operation information from the JournalNode. After existing data synchronization is complete, synchronize incremental data based on the file system change operation log.

[0084] The key challenge of process 1 lies in how to perform distributed data synchronization on each data node and how to limit the data synchronization rate to avoid impacting the online environment.

[0085] The key challenge of process 2 lies in how to replay the incremental data synchronization process based on the logs.

[0086] 2. Synchronize existing data

[0087] (1) Architecture for synchronizing existing data

[0088] Existing data synchronization employs a fairly conventional master-slave architecture, consisting of a master service and slave services. The master service acts as the central node, responsible for decomposing data synchronization tasks. Slave services are deployed on each DataNode in the Hadoop dataset, receiving data synchronization instructions from the master service and executing the synchronization process, such as... Figure 2 As shown.

[0089] (2) How does the main service control data synchronization? The workflow of the main service is as follows:

[0090] (a) Parse the backed-up NameNode data

[0091] (b) Establish the mapping relationship between DataNode nodes and offline Block replicas.

[0092] Extract the block replicas marked as offline from the backed-up NameNode metadata, and establish a mapping relationship between the DataNode nodes and the block replicas marked as offline.

[0093] One DataNode corresponds to multiple replicas of Blocks marked as offline. The mapping structure is as follows:

[0094]

[0095]

[0096] (c) Data synchronization task decomposition

[0097] The files that a DataNode needs to synchronize consist of replicas of offline blocks; that is, the files that a DataNode needs to synchronize consist of one or more replicas of offline blocks. There are two types of relationships between the files that a DataNode needs to synchronize and the replicas of offline blocks:

[0098] DataNode nodes need to synchronize files and offline block replicas: a one-to-one relationship.

[0099] DataNode nodes need to synchronize files and offline block replicas: a one-to-many relationship.

[0100] In most scenarios, the files that DataNodes need to synchronize and the offline block replicas have a one-to-one relationship. However, in a small number of DataNodes, the files that need to be synchronized consist of two or more offline block replicas.

[0101] The files that need to be synchronized by the DataNode and the offline Block replicas are one-to-one: the data synchronization task is sent to the DataNode where the offline Block replicas are located.

[0102] The files that need to be synchronized by the DataNode and the offline block replicas are in a one-to-many relationship: the data synchronization task is sent to the DataNode where the largest offline block replica is located. If there are two offline block replicas of the same size, the DataNode where the offline block replica is located is randomly selected.

[0103] The service is deployed on DataNode nodes, receives data synchronization tasks from the main service, and performs data synchronization operations.

[0104] (3) How does the service synchronize data and apply rate limiting?

[0105] Deploy a slave service on each HDFS DataNode to receive instructions from the master service, as well as information on files and offline block replicas that the current DataNode needs to synchronize. Perform the following operations:

[0106] (a) If the file that the current DataNode needs to synchronize corresponds to only one offline Block replica, it means that the offline Block replica belongs to the current DataNode. Read the data of the offline Block replica from the disk of the current DataNode and then send it to object storage.

[0107] (b) If the file that the current DataNode needs to synchronize corresponds to two or more offline Block replicas, it means that the largest offline Block replica is on the current DataNode, and the other offline Block replicas are stored on other DataNodes. In this case, the data of the offline Block replicas is read from the disk of the current DataNode, and the data of the offline Block replicas is pulled from other DataNodes. The data of these offline Block replicas are then concatenated and sent to object storage.

[0108] Since data is sent to object storage on each DataNode, a solution needs to be found to avoid limiting the data synchronization rate in order to prevent impacting online services. The specific implementation approach is as follows:

[0109] (a) On each HDFS DataNode, initialize a data synchronization queue with a fixed capacity. Elements in the data synchronization queue have no business meaning and are only used to trigger a single data synchronization request.

[0110] The synchronization of a complete file is broken down into multiple data synchronization requests. Each data synchronization request, upon initiation, retrieves one element from the data synchronization queue before proceeding. If an element is available in the queue, the request can execute immediately, removing an element from the queue. If no element is available, the request must wait until one becomes available.

[0111] (b) Data synchronization queue generation: Elements in the data synchronization queue are generated at a fixed rate, for example, N elements are generated per second, and the generated elements are placed in the data synchronization queue.

[0112] (c) Data synchronization queue capacity: The data synchronization queue has a capacity limit, which is the maximum number of elements that can be stored in the data synchronization queue. If the data synchronization queue is full and no new elements are generated, then additional data synchronization requests will not be able to obtain elements, thus limiting the data synchronization rate.

[0113] For example, if the data transmission rate is 1000 bytes / second according to the business request, and it is sent in 5 parts, then it is sent once every 200 milliseconds, and the size of the sent data is 200 bytes.

[0114] The size of the data synchronization queue can be specified as 5, which means that 5 elements of the data synchronization queue are generated per second.

[0115] When performing data synchronization, two parameters need to be specified:

[0116] The synchronization rate is the number of bytes sent per second.

[0117] The capacity N of the synchronization queue is the maximum number of data synchronization queue elements generated per second.

[0118] Based on these two parameters, the number of data synchronization requests and the amount of data sent each time can be calculated.

[0119] The number of data synchronization requests per second = N.

[0120] The amount of data sent in each synchronization request = Rate / N.

[0121] (4) Synchronization of existing data completed

[0122] After the existing data synchronization is completed, the slave services of each DataNode send messages to notify the master service. The master service monitors the data synchronization status. When the data synchronization tasks of all DataNodes are completed, the offline Block replicas are brought back online, and the incremental data synchronization operation is started.

[0123] 3. Incremental data synchronization

[0124] The incremental data synchronization operation is as follows:

[0125] (1) Data manipulation

[0126] HDFS data operations mainly include:

[0127] New data;

[0128] Delete data;

[0129] Modify the data.

[0130] All operations involving adding, modifying, or deleting data are recorded in the JournalNode's Edit Log. However, the JournalNode has a capacity limit and is periodically cleaned up.

[0131] (2) Operation of synchronizing data changes

[0132] The system synchronizes data change information from the JournalNode, extracting key operations such as adding and deleting files, adding blocks, and deleting blocks as a file system change operation log. Starting from the maximum offset of the JournalNode's Edit Log recorded during the NameNode metadata copying process, it synchronizes data change information from the JournalNode. After existing data synchronization is complete, incremental data is synchronized based on the file system change operation log.

[0133] Synchronized incremental data can be stored in a message queue Kafka or a file system.

[0134] (3) Incremental synchronization of data based on operation information of data changes

[0135] After the existing data is synchronized, start the incremental data synchronization program.

[0136] Incremental data synchronization consumes data from the message queue Kafka or file system according to the chronological order of data change operation information. Based on the CRUD (Create, Read, Update, Delete) operations, it retrieves data change operation information and performs data synchronization, as detailed below:

[0137] (a) Retrieve data change information for a batch at a time, for example, retrieve data change information for 1000 data entries at a time.

[0138] (b) Integrate operational information on data changes.

[0139] For example 1: For a file, there are two data change operation messages, which are sorted according to the time of their creation and correspond to addition and modification respectively. Since the file is ultimately modified, the data synchronization operation is simply to synchronize the modified file to object storage.

[0140] Example 2: For a file, there are three data change operation records, ordered by the time the data change operation records were generated, corresponding to the operations of addition, modification, and deletion. This file is ultimately deleted, so this file in this batch does not need to be synchronized. Therefore, the data synchronization operation is: no operation.

[0141] For example, 3: For a file, there is a single data change operation, corresponding to the file modification operation. The data synchronization operation is: delete the file in object storage and upload the modified file to object storage.

[0142] (c) Based on the content of the synchronization operation to be performed, perform data synchronization and write it to object storage.

[0143] 4. Platform Switching

[0144] During the data synchronization to object storage, the data entry, cleaning, and analysis processes can be pre-connected to object storage, which is equivalent to having two sets of programs coexisting: one connected to the original Hadoop and the other connected to object storage.

[0145] When the data synchronization progress to object storage is consistent with the data writing progress to Hadoop, all programs such as data entry, cleaning, and analysis can be connected to object storage to achieve a smooth migration.

[0146] Based on the same inventive concept, this invention also proposes an apparatus for smoothly migrating data from Hadoop to object storage. The implementation of this apparatus can refer to the implementation of the methods described above, and repeated details will not be repeated. The term "module" as used below can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0147] Figure 3 This is a schematic diagram of the device structure for smoothly migrating data from Hadoop to object storage according to the present invention. Figure 3 As shown, the device includes:

[0148] The NameNode metadata copy module 101 is used to pause Hadoop write operations, back up metadata from the primary NameNode, record the current maximum offset of the HDFS JournalNode's edit log, take down any block replica in the backed-up NameNode metadata, and resume Hadoop write operations.

[0149] The existing data synchronization module 102 is used to parse the backed-up NameNode metadata, establish the mapping relationship between DataNote nodes and offline Block replicas, decompose data synchronization tasks, synchronize data, and limit the data synchronization rate.

[0150] Data synchronization task breakdown, including:

[0151] If the files that need to be synchronized by the DataNode node and the offline Block replicas are one-to-one, the main service will send the data synchronization task to the DataNode node where the offline Block replicas are located.

[0152] If the files to be synchronized by a DataNode and the offline block replicas are in a one-to-many relationship, the main service will distribute the data synchronization task to the DataNode where the largest offline block replica is located. If there are two offline block replicas of the same size, the DataNode where the offline block replica is located will be randomly selected.

[0153] Synchronized data includes:

[0154] If the file that the current DataNode needs to synchronize corresponds to only one offline Block replica, then the slave service deployed on the DataNode reads the data of the offline Block replica from the disk of the current DataNode and sends it to object storage.

[0155] If the file that the current DataNode needs to synchronize corresponds to two or more offline Block replicas, the slave service deployed on the DataNode will read the data of the offline Block replicas from the disk of the current DataNode, pull the data of the offline Block replicas from other DataNodes, concatenate the data of these offline Block replicas, and send them to object storage.

[0156] Limiting the rate of data synchronization includes:

[0157] On each HDFS DataNode, a data synchronization queue is initialized with a fixed capacity; the elements in the data synchronization queue are used to trigger a data synchronization request.

[0158] Elements in the data synchronization queue are generated at a fixed rate, and the generated elements are placed into the data synchronization queue.

[0159] The data synchronization queue has a capacity limit. If the data synchronization queue is full and no new elements are generated, the extra data synchronization requests will not be able to obtain the elements, thus limiting the data synchronization rate.

[0160] The incremental data synchronization module 103 is used to synchronize the data change operation information recorded in the editing log of the HDFS JournalNode, starting from the current maximum offset of the edit log of the HDFS JournalNode, and incrementally synchronize data based on the data change operation information for smooth migration.

[0161] When the progress of data synchronization to object storage is consistent with the progress of data writing to Hadoop, the data entry, cleaning, and analysis programs are connected to object storage.

[0162] It should be noted that although several modules for smoothly migrating Hadoop data to object storage have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the invention, the features and functions of two or more modules described above can be embodied in a single module. Conversely, the features and functions of a single module described above can be further divided and embodied by multiple modules.

[0163] Based on the aforementioned inventive concept, such as Figure 4 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored on the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, it implements the aforementioned method for smoothly migrating Hadoop data to object storage.

[0164] Based on the aforementioned inventive concept, the present invention also proposes a computer-readable storage medium storing a computer program that executes the aforementioned method for smoothly migrating data from Hadoop to object storage.

[0165] The method and apparatus for smoothly migrating data from Hadoop to object storage proposed in this invention have the following highlights:

[0166] 1. High synchronization efficiency: Data synchronization is performed separately on each DataNode in Hadoop, resulting in high efficiency.

[0167] 2. Does not affect the operational efficiency of normal online business: Massive data synchronization tasks are distributed to each DataNode node, and the data synchronization operation rate can be limited.

[0168] 3. Supports hot migration: During data synchronization, write operations on the big data cluster are only paused briefly during the NameNode metadata copy. After the NameNode metadata copy is complete, the big data cluster continuously performs data upload, deletion, and movement operations, all of which can be synchronized to object storage.

[0169] 4. Smooth Migration: During data synchronization to object storage, data ingestion, cleaning, and analysis processes can be pre-connected to object storage. Since the data synchronization progress to object storage is synchronized with the data writing progress to Hadoop, all processes, including ingestion, cleaning, and analysis, can be seamlessly integrated into object storage, achieving a smooth migration.

[0170] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0171] Regarding the limitation of the scope of protection of this invention, those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solution of this invention are still within the scope of protection of this invention.

Claims

1. A method for smoothly migrating data from Hadoop to object storage, characterized in that, The method includes: Create a copy of NameNode metadata: Pause Hadoop write operations, back up metadata from the primary NameNode, record the current maximum offset of the HDFS JournalNode's edit log, take one copy of each block in the backed-up NameNode metadata offline, and resume Hadoop write operations. Existing data synchronization: Parse the backed-up NameNode metadata, establish the mapping relationship between DataNote nodes and offline Block replicas, decompose the data synchronization task, synchronize data and limit the data synchronization rate; Incremental data synchronization: Starting from the current maximum offset of the edit log of the HDFS JournalNode, the operation information of data change recorded in the edit log of the HDFS JournalNode is synchronized incrementally based on the operation information of data change, so as to achieve smooth migration.

2. The method for smoothly migrating data from Hadoop to object storage according to claim 1, characterized in that, When the progress of data synchronization to object storage is consistent with the progress of data writing to Hadoop, the data entry, cleaning, and analysis programs are connected to object storage.

3. The method for smoothly migrating data from Hadoop to object storage according to claim 1, characterized in that, The data synchronization task is decomposed into the following parts: If the files that need to be synchronized by the DataNode node and the offline Block replicas are one-to-one, the main service will distribute the data synchronization task to the DataNode node where the offline Block replicas are located. If the files to be synchronized by a DataNode and the offline block replicas are in a one-to-many relationship, the main service will distribute the data synchronization task to the DataNode where the largest offline block replica is located. If there are two offline block replicas of the same size, the DataNode where the offline block replica is located will be randomly selected.

4. The method for smoothly migrating data from Hadoop to object storage according to claim 1, characterized in that, The synchronized data includes: If the file that the current DataNode needs to synchronize corresponds to only one offline Block replica, then the slave service deployed on the DataNode reads the data of the offline Block replica from the disk of the current DataNode and sends it to object storage. If the file that the current DataNode needs to synchronize corresponds to two or more offline Block replicas, the slave service deployed on the DataNode will read the data of the offline Block replicas from the disk of the current DataNode, pull the data of the offline Block replicas from other DataNodes, concatenate the data of these offline Block replicas, and send them to object storage.

5. The method for smoothly migrating data from Hadoop to object storage according to claim 1, characterized in that, The limitation on the data synchronization rate includes: On each HDFS DataNode, a data synchronization queue is initialized with a fixed capacity; the elements in the data synchronization queue are used to trigger a data synchronization request. Elements in the data synchronization queue are generated at a fixed rate, and the generated elements are placed into the data synchronization queue. The data synchronization queue has a capacity limit. If the data synchronization queue is full and no new elements are generated, the extra data synchronization requests will not be able to obtain the elements, thus limiting the data synchronization rate.

6. An apparatus for smoothly migrating data from Hadoop to object storage, characterized in that, The device includes: The NameNode metadata copy module is used to pause Hadoop write operations, back up metadata from the primary NameNode, record the current maximum offset of the HDFS JournalNode's edit log, take any block copy of the backed-up NameNode metadata offline, and resume Hadoop write operations. The existing data synchronization module is used to parse the backed-up NameNode metadata, establish the mapping relationship between DataNote nodes and offline Block replicas, decompose data synchronization tasks, synchronize data, and limit the data synchronization rate. The incremental data synchronization module is used to synchronize the data change operation information recorded in the HDFS JournalNode's edit log, starting from the current maximum offset of the recorded HDFS JournalNode's edit log. Based on the data change operation information, the module incrementally synchronizes data for smooth migration.

7. The apparatus for smoothly migrating data from Hadoop to object storage according to claim 6, characterized in that, When the progress of data synchronization to object storage is consistent with the progress of data writing to Hadoop, the data entry, cleaning, and analysis programs are connected to object storage.

8. The apparatus for smoothly migrating Hadoop data to object storage according to claim 6, characterized in that, The data synchronization task is decomposed into the following parts: If the files that need to be synchronized by the DataNode node and the offline Block replicas are one-to-one, the main service will distribute the data synchronization task to the DataNode node where the offline Block replicas are located. If the files to be synchronized by a DataNode and the offline block replicas are in a one-to-many relationship, the main service will distribute the data synchronization task to the DataNode where the largest offline block replica is located. If there are two offline block replicas of the same size, the DataNode where the offline block replica is located will be randomly selected.

9. The apparatus for smoothly migrating Hadoop data to object storage according to claim 6, characterized in that, The synchronized data includes: If the file that the current DataNode needs to synchronize corresponds to only one offline Block replica, then the slave service deployed on the DataNode reads the data of the offline Block replica from the disk of the current DataNode and sends it to object storage. If the file that the current DataNode needs to synchronize corresponds to two or more offline Block replicas, the slave service deployed on the DataNode will read the data of the offline Block replicas from the disk of the current DataNode, pull the data of the offline Block replicas from other DataNodes, concatenate the data of these offline Block replicas, and send them to object storage.

10. The apparatus for smoothly migrating Hadoop data to object storage according to claim 6, characterized in that, The limitation on the data synchronization rate includes: On each HDFS DataNode, a data synchronization queue is initialized with a fixed capacity; the elements in the data synchronization queue are used to trigger a data synchronization request. Elements in the data synchronization queue are generated at a fixed rate, and the generated elements are placed into the data synchronization queue. The data synchronization queue has a capacity limit. If the data synchronization queue is full and no new elements are generated, the extra data synchronization requests will not be able to obtain the elements, thus limiting the data synchronization rate.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1-5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the method according to any one of claims 1-5.

Images