Data migration method, device and equipment based on HDFS (Hadoop Distributed File System) block state, medium and product
By setting maintenance mode and optimizing data migration strategies in HDFS, the problems of intranet traffic and data loss in HDFS data migration are solved, and operation and maintenance capabilities and business stability are improved.
Patent Information
- Application Number
- CN202411232821.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-04
AI Technical Summary
When data migration in HDFS, the existing technology will lead to a large risk of intranet traffic and data loss, affecting the stability of cluster services.
Data migration method based on HDFS block state, by setting the source disk to enter maintenance mode, counting the copy status of block files, generating a task list, and migrating block files to other disks in priority order according to the expected storage amount of other disks. Avoid missing copies and intranet traffic.
It reduces the risk of impact on cluster services, improves the operation and maintenance capabilities of HDFS disks, realizes data migration within nodes, avoids intranet traffic when replicas are missing, and ensures data reliability and stability.
Smart Images

Figure CN119088314A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data migration method, device, equipment, medium and product based on HDFS block status. Background Art
[0002] Hadoop Distributed File System (HDFS) is a core project of Hadoop. As a distributed file system, it carries the storage of the entire Hadoop ecosystem and provides scalable, high-throughput, and highly reliable data storage services for upper-level applications and users in the Hadoop ecosystem. It is suitable for distributed reading and writing of large-scale data, especially in scenarios with more reads than writes.
[0003] HDFS usually stores data on multiple disks on multiple data nodes. HDFS provides two optional solutions to uninstall disks to achieve the purpose of reducing or replacing disks. Solution 1 directly uses the decommissioning mechanism provided by HDFS to reduce the number of DataNodes; Solution 2 modifies the dfs.datanode.data.dir configuration item parameter on the DataNode that specifies the storage location of the HDFS data block and removes the disk directory to be uninstalled from its value. However, both of the above solutions will cause a large amount of data to move between nodes, generate a large amount of intranet traffic that affects cluster services, and have a high risk of data loss. Summary of the invention
[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a data migration method, device, equipment, medium and product based on HDFS block status, which can avoid generating intranet traffic to supplement the copies when the copies are missing, thereby reducing the risk of impact on cluster services.
[0005] In order to achieve the above purpose, the technical solutions provided by the embodiments of the present application are as follows:
[0006] In a first aspect, the present application provides a data migration method based on HDFS block status, including: setting a source disk to enter maintenance mode; counting the copy status of block files in the source disk and generating a task list; the task list includes block files corresponding to a preset copy status; obtaining the storage capacity and capacity of the source disk and other disks, and calculating the expected storage capacity of other disks; the source disk and the other disks belong to the same data node; when the expected storage capacity of other disks is greater than the storage capacity of the source disk, migrating the block files in the task list to other disks in priority order, and the priority order corresponds to the preset copy status.
[0007] As an optional implementation method of the embodiment of the present application, the storage amount and capacity of the source disk and other disks are obtained, and the expected storage capacity of other disks is calculated, including: calculating the total storage amount of the data node based on the storage amount of the source disk and other disks; calculating the expected total capacity of the data node based on the capacity of other disks; calculating the expected storage rate based on the total storage amount and the expected total capacity of the data node; calculating the expected storage amount of other disks based on the expected storage rate and the capacity of other disks; calculating the expected storage capacity of other disks based on the expected storage amount of other disks and the storage capacity of other disks.
[0008] As an optional implementation of the embodiment of the present application, after obtaining the storage capacity and capacity of the source disk and other disks and calculating the expected storage capacity of the other disks, the method also includes: when the expected storage capacity of the other disks is less than the storage capacity of the source disk, obtaining the reserved storage capacity of the other disks; calculating the maximum storage capacity of the other disks based on the reserved storage capacity of the other disks and the storage capacity of the other disks; when the maximum storage capacity of the other disks is greater than the storage capacity of the source disk, migrating the block files in the task list to the other disks in order of priority.
[0009] As an optional implementation of an embodiment of the present application, after calculating the maximum storage capacity of other disks based on the reserved storage capacity of other disks and the storage capacity of other disks, the method also includes: when the maximum storage capacity of other disks is less than the storage capacity of the source disk, determining whether to migrate across nodes; if so, migrating the block files in the task list to other disks in order of priority until the storage capacity of other disks reaches the maximum storage capacity, and then migrating the remaining block files across nodes to the disks of other data nodes.
[0010] As an optional implementation of an embodiment of the present application, the preset replica status includes: a first replica status, indicating that the replica is in the Finalized status where writing has been completed and an additional write operation can be performed; a second replica status, indicating that the replica is in the Finalized status and no additional write operations are performed; a third replica status, indicating that the replica is in the writing status.
[0011] As an optional implementation of the embodiment of the present application, when the expected storage capacity of other disks is greater than the storage capacity of the source disk, the block files in the task list are migrated to other disks in order of priority, including: when the expected storage capacity of other disks is greater than the storage capacity of the source disk, the block files corresponding to the first copy state are migrated to other disks, and then the block files corresponding to the second copy state are migrated to other disks; for the block files corresponding to the third copy state, after a preset waiting time, the target block files are migrated to other disks; wherein the target block files are block files converted to the Finalized state among the block files corresponding to the third copy state.
[0012] In a second aspect, the present application provides a data migration device based on HDFS block status, the device comprising:
[0013] The setting module is used to set the source disk to enter the maintenance mode;
[0014] A statistics module is used to count the copy status of block files in the source disk and generate a task list; the task list includes block files corresponding to the preset copy status;
[0015] A calculation module is used to obtain the storage amount and capacity of the source disk and other disks, and calculate the expected storage amount of other disks; the source disk and other disks belong to the same data node;
[0016] The migration module is used to migrate the block files in the task list to other disks in a priority order when the expected storage capacity of other disks is greater than the storage capacity of the source disk, and the priority order corresponds to the preset copy state.
[0017] As an optional implementation manner of the embodiment of the present application, the calculation module is specifically used to: calculate the total storage amount of the data node based on the storage amount of the source disk and other disks; calculate the expected total capacity of the data node based on the capacity of other disks; calculate the expected storage rate based on the total storage amount of the data node and the expected total capacity; calculate the expected storage amount of other disks based on the expected storage rate and the capacity of other disks; calculate the expected storage capacity of other disks based on the expected storage amount of other disks and the storage capacity of other disks.
[0018] As an optional implementation of the embodiment of the present application, the calculation module is also used to: when the expected storage capacity of other disks is less than the storage capacity of the source disk, obtain the reserved storage capacity of other disks; calculate the maximum storage capacity of other disks based on the reserved storage capacity of other disks and the storage capacity of other disks; the migration module is also used to migrate the block files in the task list to other disks in priority order when the maximum storage capacity of other disks is greater than the storage capacity of the source disk.
[0019] As an optional implementation of the embodiment of the present application, the computing module is also used to: when the maximum storage capacity of other disks is less than the storage capacity of the source disk, determine whether to migrate across nodes; the migration module is also used to: if so, migrate the block files in the task list to other disks in order of priority until the storage capacity of other disks reaches the maximum storage capacity, and then migrate the remaining block files across nodes to the disks of other data nodes.
[0020] As an optional implementation of an embodiment of the present application, the preset replica status includes: a first replica status, indicating that the replica is in the Finalized status where writing has been completed and an additional write operation can be performed; a second replica status, indicating that the replica is in the Finalized status and no additional write operations are performed; a third replica status, indicating that the replica is in the writing status.
[0021] As an optional implementation of the embodiment of the present application, the migration module is specifically used to: when the expected storage capacity of other disks is greater than the storage capacity of the source disk, migrate the block files corresponding to the first copy state to other disks, and then migrate the block files corresponding to the second copy state to other disks; for the block files corresponding to the third copy state, after a preset waiting time, migrate the target block files to other disks; wherein the target block files are block files converted to the Finalized state among the block files corresponding to the third copy state.
[0022] In a third aspect, the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the data migration method based on the HDFS block state as described in the first aspect or any one of its optional embodiments is implemented.
[0023] In a fourth aspect, the present application provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the data migration method based on HDFS block status as described in the first aspect or any one of its optional embodiments.
[0024] In a fifth aspect, the present application provides a computer program product, including: the computer program product includes a computer program, and when the computer program runs on a computer, the computer implements the data migration method based on HDFS block status as described in the first aspect or any one of its optional embodiments.
[0025] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:
[0026] The disclosed embodiment provides a data migration method based on the HDFS block status, firstly setting the source disk to enter the maintenance mode, then counting the copy status of the block files in the source disk, generating a task list consisting of block files corresponding to the preset copy status; then obtaining the storage capacity and capacity of the source disk and other disks on the data node, calculating the expected storage capacity of other disks, and further, in the case that the expected storage capacity of other disks is greater than the storage capacity of the source disk, the block files in the task list are migrated to other disks according to the priority order corresponding to the preset copy status. In this way, the present application sets the maintenance mode at the disk level to avoid adverse effects on other operations of the data node; by calculating the expected storage capacity of other disks, measuring whether other disks can carry the source disk data, the operation and maintenance capabilities of the HDFS disk are improved, data migration within the node is realized, and the generation of intranet traffic to supplement the copy when the copy is missing is avoided, thereby reducing the risk of cluster services being impacted. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0029] Figure 1A This is a diagram of the directory structure of a data node on a local disk;
[0030] Figure 1B This is a pseudo code diagram of the HDFS replica status;
[0031] Figure 2 A flowchart of a data migration method based on HDFS block status according to an embodiment of the present application;
[0032] Figure 3A A pseudo code schematic diagram for setting a source disk to enter maintenance mode provided in an embodiment of the present application;
[0033] Figure 3B A schematic diagram of parameters for setting a source disk to enter maintenance mode provided in an embodiment of the present application;
[0034] Figure 3C A schematic diagram of an execution command for setting a source disk to enter maintenance mode provided in an embodiment of the present application;
[0035] Figure 4AA schematic diagram of the parameters, default values and descriptions included in the Plan command provided in an embodiment of the present application;
[0036] Figure 4B A schematic diagram of the parameters, default values and descriptions included in the Execute command provided in the embodiment of the present application;
[0037] Figure 4C A schematic diagram of the parameters, default values and descriptions included in the Query command provided in the embodiment of the present application;
[0038] Figure 4D A pseudo code diagram of the Cancel command provided in an embodiment of the present application;
[0039] Figure 5 A schematic diagram of the structure of a data migration device based on HDFS block status provided in an embodiment of the present application;
[0040] Figure 6 A schematic diagram of the structure of an electronic device described in an embodiment of the present application. DETAILED DESCRIPTION
[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the technical terms required to be used in the embodiments or the prior art description are briefly introduced below:
[0042] HDFS adopts a master-slave architecture, consisting of NameNode (NN) and DataNode. NameNode is the master node, which is mainly responsible for managing the HDFS namespace, parsing client requests, and controlling client access to HDFS. DataNode is a slave node, mainly used for physical storage, and its data volume depends on the size of the cluster.
[0043] NameNode is the master node of HDFS. Its main functions are: responsible for managing the HDFS namespace, cluster information and data blocks; maintaining the entire HDFS file directory tree, the metadata of the file directory and the list of data blocks corresponding to each file; accepting client operation requests; managing the mapping relationship between files and data blocks, and between data blocks and DataNodes.
[0044] DataNode is responsible for processing read and write requests from file system clients. It creates, copies, and deletes data blocks under the unified scheduling of NameNode. DataNode encapsulates a structure in memory: the data storage component (FsDataset), which is responsible for dynamically maintaining all data block (Block) sets on this node. Because Block files exist in various storage disks and directories, it is also responsible for maintaining the storage directory (i.e., disk). In DataNode, each storage directory is encapsulated into a volume (FsVolume), simulating the "mount point" under the Linux system. Under normal circumstances, FsVolume can be considered as an abstraction of the disk. Sometimes it is necessary to access the real disk (such as retrieving disk available space), so it is necessary for the abstraction to interact directly with the real disk, which is mainly implemented by calling respective Linux commands in Java.
[0045] In HDFS, Block, i.e. its replica data, is the basic storage unit of a file. Files are connected in an orderly and regular manner through Block file blocks. Replica data can be understood as the specific storage embodiment of Block. HDFS configures the size of Block through ${dfs.blocksize}, which is 128MB by default. When a new Block is generated, the replica location determines the storage location of the replica in the cluster, which is determined by the location selection strategy and storage strategy.
[0046] Maintenance mode, Maintenance state also refers to a new state of data nodes in HDFS (Hadoop Distributed File System), which is designed to reduce the impact of cluster maintenance. In this state, the node will not trigger replication operations, will not accept read and write requests, and will not be selected as a storage node, allowing maintenance without affecting data availability. The basic purpose of this state is to allow nodes to temporarily leave the cluster without transferring the data on them, while ensuring the basic availability of the cluster and the accessibility of the data. This state is suitable for operations such as rolling upgrades and fast switching of services to deal with software and hardware failures.
[0047] The purpose of entering the Maintenance state is to ensure data availability, especially when the data node has the only copy of certain data in the cluster. Compared with the Decommission process, the purpose of the Maintenance state is to reduce the impact on the cluster, because it focuses on ensuring the existence of at least one copy of the data, rather than the Decommission process, which may require the transfer of data to meet the copy coefficient requirements. Therefore, in the Maintenance state, unless there is no copy of the data on other nodes, there will be basically no data transfer operations, thereby reducing the impact of maintenance operations on the cluster. In general, the HDFS Maintenance state provides a mechanism that allows nodes to perform necessary maintenance work without seriously affecting the availability and integrity of cluster data, while ensuring the stability and reliability of the system. It should be noted that Figure 1A This is just an example of the directory structure of a local disk on a data node. The complete directory structure of a data node is shown in the following figure. Figure 1A Not shown in full.
[0048] The directory structure of the disk. The DataNode volumes correspond to the directories configured by dfs.datanode.data.dir, which is where the HDFSDataNode stores the blocks. It can be a comma-separated list of directories (typically, each directory is on a different disk). Figure 1AAs shown in the figure, the block pool directory (BlockPoolSlice), which starts with BP-xxxx, is the directory of a block pool (BlockPool), such as BP-880450491-10.0.0.8-1721802325640. Under the pool block directory, the current directory is the actual entry for this BlockPool to access data in this disk; scanner.cursor file: DataNode will regularly verify each blk file, and this file is used to record the location of the verification. The tmp directory is used to store some temporary operation data. Under the current directory, dfsUsed saves the disk usage statistics, which are refreshed every 10 minutes. The Finalized directory is used to store the Block files (such as blk_1073741836) and their validation files (such as blk_1073741836_1012.meta) that have been written. The validation file ends with meta, which is the metadata file of the Block, which saves the verification information, meta length, etc. The rbw directory stores the Block files being written, which will be moved to the Finalized directory after writing is completed; the VERSION file is used to record information related to Namespace and BlockPool.
[0049] The replica state is maintained by the DataNode node, which mainly displays the state and visibility of the current replica data. The state will always be there from the time the data arrives until it is deleted. This state also directly affects the Block state on the NameNode side. The replica state is defined in ReplicaState. The pseudo code of the replica state is as follows: Figure 1B shown.
[0050] The replica status includes: Replica Being Written (RBW), Replica is Finalized (Finalized), Replica Waiting to be Recovered (RWR), Replica is Under Recovery (RUR), and Temporary.
[0051] RBW, from the moment the DataNode creates a new replica locally and starts receiving Block data, the replica is in the RBW state. Data in this state will be temporarily stored in the current / BP-xxx / current / rbw directory. Once all the data of the Block is received, the data will be moved from the rbw directory to the Finalized directory, a process that will be handled by the DataNode itself. Data in this state will be quickly reported to the NameNode so that the NameNode can further process the Block based on the replica status.
[0052] Finalized: For a Block, at least one replica must be in the Finalized state to indicate that the Client has completed the update of the Block. In actual use, DataNode stores the most replica data in the Finalized state, so DataNode also uses more mechanisms to ensure efficient access and prevent such data from being lost. Finalized is implemented through FinalizedReplica. The replica in this state will save the corresponding checksum value, which is the value of blk_xxx.meta introduced earlier.
[0053] RWR, when DataNode goes down, the replica data being written will be different from the replicas on other nodes. When the service is restarted, part of the replica data in current / BP-xxxx / current / rbw needs to be restored to maintain consistency between different replicas of the same Block.
[0054] RUR, evolved from RWR, indicates that an existing replica is being recovered if in this state. Recovery means redefining the data content and generationStamp of the current replica to keep it consistent with the data of other replicas belonging to the same block. In the RWR and RUR states, the numBytes and generationStamp values related to the replica are active and need to wait for the entire recovery process to be completed before they can be determined.
[0055] Temporary, the corresponding data directory is current / BP-xxxx / tmp. The replicas of this state are special, mainly storing data scheduled within the cluster. Usually there are two situations where this type of data is generated: (1) Balance operation within the cluster. When performing cluster-level load balancing, the target data node temporarily places the received data in the tmp directory, and after all data is successfully received, it will be moved to the Finalized directory. (2) Replica data completion. When performing replica data completion on a Block, the received data will also be temporarily placed here, and after successful completion, it will be moved to the Finalized directory.
[0056] Block Pool Directory ( Figure 1A There are subdirectories current and tmp under BP-880450491-10.0.0.8-1721802325640. The tmp directory is used to store copies of the Temporary state. The Finalized directory under the current directory stores copies of the Finalized state, and the rbw directory stores copies of the RBW, RWR, and RUR states.
[0057] It can be seen that all data files are stored in the Finalized directory and the rbw directory, and there will be no data files with the same Block ID on the same DataNode. Therefore, the data on one disk can be moved to another disk by migrating the blk file, and then the disk can be uninstalled to achieve the purpose of shrinking.
[0058] There are currently two solutions for reducing the storage capacity of an HDFS cluster: The first is to reduce the number of DataNode nodes. If there are a large number of DataNode nodes, and it is possible to reduce the CPU and memory resources while reducing the storage space, you can reduce several DataNode nodes and directly use the decommission provided by HDFS. This is the most common method. The reduction process involves the cross-node migration of a large amount of data, which will generate a large amount of intranet traffic and may affect the online load. Operation and maintenance personnel need to pay close attention and manually tune. If there are only three DataNode nodes left in the cluster, or the CPU or memory resources above cannot be reduced synchronously, this method cannot be used.
[0059] The second method is to reduce the number of disk directories on a single DataNode. That is, while keeping the number of DataNode nodes unchanged, reduce the disk space on each node. You can modify the dfs.datanode.data.dir parameter on the DataNode, delete one or more disk directories, and then wait for HDFS to automatically add copies. This method will also cause a large amount of data to move between nodes, generate a large amount of intranet traffic, and may affect the online load. Operation and maintenance personnel need to pay close attention and manually tune. In addition, if there are only two copies of the data, it will be relatively dangerous. Once a node has a problem or a disk fails when a disk directory is deleted, data loss is very likely to occur.
[0060] Disk replacement usually involves replacing a disk with a longer service life with a new disk with more stable performance, or replacing a disk with a smaller storage specification with a larger disk. During long-term operation, some hard disks may experience longer delays and slower read and write times due to the disk itself or frequent read and write operations. The "slow disk" phenomenon is common in distributed storage architectures: when a disk has a performance failure or the IO load is too high, causing the delay to increase, almost all user services will be seriously affected. The "slow disk phenomenon" will cause the performance of the entire cluster to degrade, and in severe cases, it may even cause data loss, service unavailability, and other problems. The slow disk phenomenon is exactly the same as the ancient barrel effect: the barrel effect refers to the maximum amount of water that a barrel can hold, which is not determined by the tallest plank, but by the shortest plank. Similarly, the performance of distributed storage is not determined by the fastest disk, but by the slowest disk.
[0061] Disk replacement method: Format and mount the hard disk to be added, modify dfs.datanode.data.dir in the configuration file hdfs-site.xml on the DataNode node, and set it to the directory to be used from now on (add a new mount point and delete the obsolete mount point). Configure the new disk to remove the old disk. Similar to the second solution of shrinking, waiting for HDFS to automatically supplement the replica will also cause a large amount of data to move between nodes.
[0062] In order to solve some or all of the technical problems existing in the above-mentioned related technologies, the embodiments of the present application provide a data migration method, device, equipment, medium and product based on the HDFS block state, wherein the method first sets the source disk to enter the maintenance mode, then counts the copy status of the block files in the source disk, and generates a task list consisting of block files corresponding to the preset copy status; then obtains the storage capacity and capacity of the source disk and other disks on the data node, calculates the expected storage capacity of other disks, and further, when the expected storage capacity of other disks is greater than the storage capacity of the source disk, the block files in the task list are migrated to other disks according to the priority order corresponding to the preset copy status. In this way, the present application sets the maintenance mode at the disk level to avoid adverse effects on other operations of the data node; by calculating the expected storage capacity of other disks, it measures whether other disks can carry the source disk data, improves the operation and maintenance capabilities of the HDFS disk, realizes data migration within the node, and avoids the generation of intranet traffic to supplement the copy when the copy is missing, thereby reducing the risk of cluster services being impacted.
[0063] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0064] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.
[0065] A data migration method based on HDFS block status provided in the embodiments of the present application can be implemented by a data migration device or electronic device based on HDFS block status, and the electronic device includes but is not limited to a vehicle terminal, a server, a personal computer, a laptop computer, a tablet computer, a smart phone, etc. The operating system of the electronic device may include Android, a mobile operating system (iOS) developed by Apple, an operating system (Windows) developed by Microsoft Corporation of the United States, etc., and the embodiments of the present application are not limited to this. The electronic device can be run alone to implement the present application, or it can be connected to the network and implement the present application through interactive operations with other computer devices in the network. Among them, the network where the electronic device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN) network, etc.
[0066] It should be noted that the protection scope of the data migration method based on HDFS block status described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.
[0067] The method provided in the embodiment of the present application is applicable to disk shrinking and disk replacement scenarios. Disk shrinking can be to uninstall a disk and copy the data on it to other disks on the same data node. Disk replacement is to first copy the data on a disk to a newly added disk on the same data node, and then uninstall the disk.
[0068] like Figure 2 As shown, Figure 2 The flowchart of a data migration method based on HDFS block status according to an embodiment of the present application is shown in FIG. 1 , which can be executed by a data migration device based on HDFS block status, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 2 As shown, the method mainly includes the following steps S201 to S204:
[0069] S201. Set the source disk to enter maintenance mode.
[0070] The source disk (volume) is the disk that needs to be unmounted and the data on it needs to be migrated. It can be regarded as the data source and is therefore defined as the source disk.
[0071] For example, Figure 3A As shown, modify the org.apache.hadoop.hdfs.server.datanode.datanode code and add the configuration item DFS_DATANODE_DATA_DIR_IN_MAINTENANCE_KEY. Figure 3B As shown in the figure, add and modify the configuration item dfs.datanode.data.dir.in.maintenance in the hdfs-size.xml file to set a disk in the data node to enter maintenance mode. Figure 3C As shown in the figure, after adding and modifying the configuration items, the command hdfsdfsadmin-reconfigdatanodeip:portstart is used to dynamically refresh the data node configuration, where ip is the data node to which the source disk belongs. It should be noted that the above modification is only for the data node, and the NameNode side is not aware of it, and does not affect the data node's external services. Compared with the community code's ability to set the node to enter maintenance mode, setting the disk-level maintenance mode can achieve more fine-grained disk-level operation and maintenance.
[0072] Step S201 sets a disk of a single data node into maintenance mode without affecting write and read operations in the node. Set the source disk into maintenance mode, skip the source disk when adding new data, so that the source disk retains the existing block files and no longer adds new block files, which can avoid data loss to a certain extent.
[0073] S202: Count the copy status of block files in the source disk and generate a task list.
[0074] The task list includes block files corresponding to the preset replica status.
[0075] It should be noted that the replica state (ReplicaState) of the block file in the source disk includes the following three types:
[0076] (1) The directory current / Finalized is in the Finalized state;
[0077] This state includes the following different forms:
[0078] ① When the preset size is reached, no additional write (append) operations are performed; the preset size is configured through ${dfs.blocksize}, and the default value for Hadoop 2.x and later versions is usually 128MB.
[0079] ② If the preset size is not reached, the append operation can be performed.
[0080] ③ In Finalized state but waiting to be deleted; for example, a Finalized blk file that needs to be deleted due to a TRUNCATE operation. This application does not consider this form.
[0081] (2) The current / rbw directory is in the RBW, RWR, or RUR state;
[0082] (3) The tmp directory is in Temporary state.
[0083] In the embodiment of the present application, the preset replica status includes: a first replica status, indicating that it is in the Finalized status and can perform additional write operations, a second replica status, indicating that it is in the Finalized status and no longer performs additional write operations, and a third replica status, indicating that it is in the replica writing status. Optionally, the third replica status includes: a replica writing status under the subdirectory tmp or current / rbw of the block pool directory, or a replica writing status of rebalancing or patching under the subdirectory tmp of the block pool directory and a replica writing status under the subdirectory current / rbw of the block pool directory. current / rbw refers to the rbw directory under the subdirectory current of the block pool directory.
[0084] Correspondingly, the task list includes: block files corresponding to the first replica state, block files corresponding to the second replica state, and block files corresponding to the third replica state. That is, block files in the Finalized state that can perform additional write operations, block files in the Finalized state that no longer perform additional write operations, block files in the subdirectory tmp or current / rbw of the block pool directory that are in the replica writing state, or block files in the subdirectory tmp of the block pool directory that are in the replica writing state of rebalancing or patching, and block files in the subdirectory current / rbw of the block pool directory that are in the replica writing state.
[0085] Exemplarily, the task list includes: blk files and their blk_xxx.meta files that have reached the configured size of ${dfs.blocksize} and will not be written again; blk files that are smaller than the configured size of ${dfs.blocksize}; block files in the subdirectory tmp or current / rbw of the block pool directory whose replicas are being written, or block files in the subdirectory tmp of the block pool directory whose replicas are being written due to rebalancing or patching, and block files in the subdirectory current / rbw of the block pool directory whose replicas are being written.
[0086] In step S202, each block pool directory in the source disk is scanned, the copy status is collected and sorted, the block files are classified according to the preset copy status, and a task list is generated.
[0087] S203: Obtain the storage capacity and capacity of the source disk and other disks, and calculate the expected storage capacity of other disks.
[0088] In some embodiments, the data node includes multiple disks, and in addition to the source disk to be uninstalled, there is at least one other disk. The other disks can be the original disks of the data node. For example, in the original state, the data node includes three disks: disk A, disk B, and disk C. Assume that disk C is the source disk to be uninstalled, and the other disks include disk A and disk B.
[0089] Other disks are disks other than the source disk on the data node, including original disks other than the source disk, and newly added disks pre-mounted to the data node. For example, in the original state, the data node includes 3 disks: Disk A, Disk B, and Disk C. Assume that Disk C is the source disk to be uninstalled and will be replaced by the newly added disk D. Other disks include Disk A, Disk B, and Disk D. The mount path of the newly added disk can be increased through the configuration item dfs.datanode.data.dir in the configuration file hdfs-site.xml. The storage capacity of the source disk indicates the amount of data that needs to be migrated, which is the data amount of all block files in the task list.
[0090] In some embodiments, when calculating the expected storage capacity of other disks, the calculation process is as follows: calculate the total storage capacity of the data node based on the storage capacity of the source disk and other disks; calculate the expected total capacity of the data node based on the capacity of other disks; calculate the expected storage rate based on the total storage capacity and the expected total capacity of the data node; calculate the expected storage capacity of other disks based on the expected storage rate and the capacity of other disks; calculate the expected storage capacity of other disks based on the expected storage capacity and storage capacity of other disks. Among them, the expected storage capacity is a storage capacity to ensure the performance of the disk.
[0091] For example, in a disk shrinking scenario, the storage capacity of disk A is 0.4TB and the capacity is 2TB; the storage capacity of disk B is 0.8TB and the capacity is 2TB; the storage capacity of disk C is 0.6TB and the capacity is 1T. Disk C is the source disk and needs to be uninstalled. First, according to the storage capacity of each disk, the total storage capacity of the data node is calculated to be 1.8TB; according to the capacity of other disks except the source disk, the total capacity of the data node is calculated to be 4TB; according to the total storage capacity and total capacity of the data node, the expected storage rate of the disk is calculated to be 0.45; according to the expected storage rate and the capacity of other disks, the expected storage capacity of other disks is calculated, and the expected storage capacity of disk A is 0.9TB, the expected storage capacity of disk B is 0.9TB, and the expected storage capacity of disk C needs to be uninstalled, and the expected storage capacity is 0; according to the expected storage capacity and current storage capacity of other disks, the expected storage capacity of other disks is calculated, and the expected storage capacity of disk A is 0.5TB, the expected storage capacity of disk B is 0.1TB, and the expected storage capacity of disk C needs to be uninstalled, and the expected storage capacity is 0. As shown in Table 1.
[0092] Table 1
[0093]
[0094] For example, in a disk replacement scenario, disk A has a storage capacity of 0.4 TB and a capacity of 2 TB; disk B has a storage capacity of 0.8 TB and a capacity of 2 TB; disk C has a storage capacity of 0.6 TB and a capacity of 1 TB. Disk C is the source disk and needs to be uninstalled. A new disk D is added to the data node, with a capacity of 2 TB. First, based on the storage capacity of each disk, the total storage capacity of the data node is calculated to be 1.8TB; based on the capacity of other disks except the source disk, the total capacity of the data node is calculated to be 6TB; based on the total storage capacity and total capacity of the data node, the expected storage rate of the disk is calculated to be 0.3; based on the expected storage rate and the capacity of other disks, the expected storage capacity of other disks is calculated, and the expected storage capacity of disk A is 0.6TB, the expected storage capacity of disk B is 0.6TB, and the expected storage capacity of disk D is 0.6TB. If disk C needs to be uninstalled, the expected storage capacity is 0; based on the expected storage capacity and current storage capacity of other disks, the expected storage capacity of other disks is calculated, and the expected storage capacity of disk A is 0.2TB, the expected storage capacity of disk B is -0.2TB, and the expected storage capacity of disk D is 0.6TB. If disk C needs to be uninstalled, the expected storage capacity is 0. As shown in Table 2.
[0095] Table 2
[0096]
[0097] The above embodiment obtains the storage amount and capacity of each disk on the data node to clarify the used space size and actual space size of each disk. If the source disk needs to be uninstalled, the space of the source disk is no longer used, the total storage amount of the data node remains unchanged but the total capacity becomes smaller, and by calculating the expected storage capacity of other disks, it is determined whether other disks on the data node can carry the data on the source disk, so as to improve the operation and maintenance capabilities of the HDFS disk.
[0098] In some embodiments, the reserved storage capacity of other disks may be obtained, and the maximum storage capacity may be calculated in combination with the capacity of other disks, so as to determine whether the storage capacity of the source disk is less than or equal to the maximum storage capacity of other disks.
[0099] The reserved storage capacity can be set through the configuration item (dfs.disk.replacer.max.diskreserved or dfs.disk.replacer.max.diskreserved.pct). The maximum storage capacity is the upper limit of the disk. The storage capacity of the disk can exceed the expected storage capacity, but cannot exceed the maximum storage capacity.
[0100] Exemplarily, the reserved storage capacity of disk A is 0.2 TB, the capacity is 2 TB, and the maximum storage capacity of disk A is calculated to be 1.8 T.
[0101] The above embodiment further obtains the reserved storage capacity of other disks on the data node except the source disk, calculates the maximum storage capacity of other disks, and clarifies the maximum amount of data that other disks can carry, so as to determine whether other disks on the data node can completely carry the data on the source disk based on the maximum storage capacity.
[0102] S204: When the expected storage capacity of other disks is greater than the storage capacity of the source disk, migrate the block files in the task list to other disks in order of priority.
[0103] Among them, the priority order corresponds to the preset replica state. Optionally, the first replica state has the highest priority, the second replica state has the second highest priority, and the third replica state has the lowest priority. That is, the state that is in the Finalized state and can perform additional write operations corresponds to the highest priority; the state that is in the Finalized state and no longer performs additional write operations has the second highest priority; the state that is in the RBW state in the rbw or tmp directory has the lowest priority.
[0104] When the expected storage capacity of other disks is greater than the storage capacity of the source disk, it means that, except for the source disk that needs to be uninstalled, other disks on the current data node have enough available space to store the data of the source disk, and other disks can be used reasonably. Then, the block files in the preset replica state that have been sorted on the source disk are migrated to other disks in order of priority to complete the data migration.
[0105] Exemplarily, when the expected storage capacity of other disks is greater than the storage capacity of the source disk, the block files that are in the Finalized state and can perform additional write operations are first migrated to other disks, and then the block files that are in the Finalized state and no longer perform additional write operations are migrated to other disks. Finally, the block files in the replica writing state under the subdirectory tmp or current / rbw of the block pool directory, or the block files in the replica writing state under the subdirectory tmp of the block pool directory that are in the rebalancing or patching state and the block files in the replica writing state under the subdirectory current / rbw of the block pool directory are migrated to other disks.
[0106] In some embodiments, when executing step S204, if the expected storage capacity of other disks is greater than the storage capacity of the source disk, the block files corresponding to the first copy state are first migrated to the other disks, and then the block files corresponding to the second copy state are migrated to the other disks; then, for the block files corresponding to the third copy state, after a preset waiting time, the target block files are migrated to the other disks; wherein the target block files are block files converted to the Finalized state among the block files corresponding to the third copy state.
[0107] Specifically, when the expected storage capacity of other disks is greater than the storage capacity of the source disk, the block files that are in the Finalized state and can perform additional write operations are migrated to other disks first in order of priority, and then the block files that are in the Finalized state and no longer perform additional write operations are migrated to other disks; then the block files that were previously in the write state in the tmp or current / rbw directory and have now been converted to the Finalized state are migrated to other disks; for the block files in the write state in the tmp or current / rbw directory, the block files that have completed writing and converted to the Finalized state after waiting for the preset waiting time (the default is 15 minutes) will be migrated to other disks, and the block files that have not completed writing after the preset waiting time will be left on the source disk.
[0108] In some embodiments, when executing step S204, there are multiple other disks, and the block files will be split and migrated to multiple other disks. To avoid unbalanced disk storage, tools (such as DiskBalancer) can be used to balance the disks. As shown in Table 2, 0.2TB of data on disk B can be migrated to disk A.
[0109] The above embodiment is applied in the disk replacement scenario, the data node can work normally, and the read operation of the source disk will not be affected; the new disk can be put into use immediately to share the storage and read and write pressure, avoiding the impact and influence of a large amount of block filling traffic on the cluster business due to the loss of replicas.
[0110] In some embodiments, after executing step S203, if the expected storage capacity of other disks is less than the storage capacity of the source disk, the reserved storage capacity of other disks is obtained, and the maximum storage capacity of other disks is calculated in combination with the storage capacity of other disks; it is determined whether the maximum storage capacity of other disks is greater than the storage capacity of the source disk. If so, step S204 is executed to migrate the block files in the task list to other disks in order of priority.
[0111] Exemplarily, it is assumed that the data node only includes disk A and disk C, and disk C needs to be uninstalled. After the above steps, it is determined that the expected storage capacity of disk A, 0.9TB, is less than the storage capacity of the source disk, 1TB, indicating that disk A may not be in an ideal state if used as a target disk. Then, the reserved storage capacity of disk A, 0.2TB, is obtained. Combined with the storage capacity of disk A, 2TB, the maximum storage capacity of disk A, 1.8TB, is calculated. Then, it is determined whether the maximum storage capacity of disk A is greater than the storage capacity of the source disk to estimate whether disk A can accommodate all the source disk data; if it is determined that 1.8TB is greater than the storage capacity of the source disk, 1TB, it means that although disk A cannot be in an ideal state after accommodating all the source disk data, it can still meet the needs of uninstalling disk C for shrinking. Then, all block files in the task list can be migrated to disk A in priority order.
[0112] When it is determined that the maximum storage capacity of other disks is greater than the storage capacity of the source disk, it means that other disks are sufficient to store the data on the source disk. The block files in the task list corresponding to the previously sorted source disk can be migrated to other disks in order of priority.
[0113] The above embodiment calculates the expected storage capacity and maximum storage capacity of other disks to ensure that the data storage capacity on other disks after data migration does not exceed the maximum storage capacity, and even does not exceed the expected storage capacity. Therefore, when the source disk is uninstalled or replaced, the normal operation of other disks is not affected, and the cluster business will not be adversely affected.
[0114] When it is determined that the maximum storage capacity of other disks is less than the storage capacity of the source disk, it means that the space of other disks on the data node except the source disk is insufficient to save the data on the source disk. In an embodiment of the present application, an optional implementation is provided, including: when the maximum storage capacity of other disks is less than the storage capacity of the source disk, determine whether to enable cross-node migration; if so, migrate the block files in the task list to other disks in order of priority until the storage capacity of other disks reaches the maximum storage capacity, and then migrate the remaining block files to the disks of other data nodes.
[0115] Specifically, when the disk in a single node does not have enough storage capacity to store the uninstalled disk, first query the cross-node parameter (crossdn) to determine whether to enable cross-node migration; if the cross-node parameter indicates to enable, first complete the data migration between disks in the node, and then migrate the remaining block files to the disks of other data nodes.
[0116] After completing the data migration through the above embodiment, the source disk is set to exit the maintenance mode, and the configuration item dfs.datanode.data.dir in the configuration file hdfs-site.xml is modified to remove the source disk from the data node.
[0117] In the above embodiment, data migration occurs inside the data node, and cross-node traffic will not be generated, thus avoiding impact and influence on cluster services.
[0118] In some embodiments, the data migration method provided in the embodiments of the present application can be executed through a command line. After setting the source disk to enter maintenance mode, execute the following command:
[0119] Run the Plan command: hdfs diskreplacer-plan datanodename to generate an operation plan. Figure 4A As shown, Figure 4A Shows the parameters, default values, and descriptions included in the Plan command.
[0120] Then execute the Execute command: hdfs diskreplacer-execute<JSON file path> , migrate block files in order of priority. Figure 4B As shown, Figure 4B Shows the parameters, default values, and descriptions of the Execute command.
[0121] Then execute the query command hdfs diskreplacer-query <datanode>, obtain the current execution progress of the HDFS disk replacer from the data node running the plan to query the data migration progress, such as Figure 4C As shown, Figure 4C Shows the parameters, default values, and descriptions included in the Query command.
[0122] After the data migration is complete, set the source disk to exit maintenance mode and modify the configuration to uninstall the source disk.
[0123] During the process, you can execute the Cancel command hdfs diskreplacer-cancel<JSON file path> , cancel the operation plan. Figure 4D As shown, Figure 4D Shows the definition of the Cancel command.
[0124] The execution of the preceding command line requires adding diskreplacer-related configuration items to the hdfs-site.xml file, as shown in Table 3.
[0125] Table 3
[0126]
[0127] The above embodiment completes data migration through the operation instructions provided by the command line, and can also intuitively feedback the execution progress, which is convenient for operation and maintenance personnel to operate and realize refined storage management at the single-node disk level.
[0128] In summary, the embodiment of the present application provides a data migration method based on the HDFS block status, which first sets the source disk to enter the maintenance mode, then counts the copy status of the block files in the source disk, and generates a task list consisting of block files corresponding to the preset copy status; then obtains the storage capacity and capacity of the source disk and other disks on the data node, calculates the expected storage capacity of other disks, and further, when the expected storage capacity of other disks is greater than the storage capacity of the source disk, migrates the block files in the task list to other disks according to the priority order corresponding to the preset copy status. In this way, the present application sets the maintenance mode at the disk level to avoid adverse effects on other operations of the data node; by calculating the expected storage capacity of other disks, it measures whether other disks can carry the source disk data, improves the operation and maintenance capabilities of the HDFS disk, realizes data migration within the node, and avoids the generation of intranet traffic to supplement the copy when the copy is missing, thereby reducing the risk of cluster services being impacted.
[0129] like Figure 5 As shown, Figure 5 A data migration device based on HDFS block status is provided in an embodiment of the present application, and the device includes:
[0130] The setting module 501 is used to set the source disk to enter the maintenance mode;
[0131] A statistics module 502 is used to count the copy status of block files in the source disk and generate a task list; the task list includes block files corresponding to the preset copy status;
[0132] The calculation module 503 is used to obtain the storage amount and capacity of the source disk and other disks, and calculate the expected storage amount of other disks; the source disk and other disks belong to the same data node;
[0133] The migration module 504 is used to migrate the block files in the task list to other disks in a priority order when the expected storage capacity of other disks is greater than the storage capacity of the source disk. The priority order corresponds to the preset copy status.
[0134] As an optional implementation of the embodiment of the present application, the calculation module 503 is specifically used to: calculate the total storage amount of the data node based on the storage amount of the source disk and other disks; calculate the expected total capacity of the data node based on the capacity of other disks; calculate the expected storage rate based on the total storage amount of the data node and the expected total capacity; calculate the expected storage amount of other disks based on the expected storage rate and the capacity of other disks; calculate the expected storage capacity of other disks based on the expected storage amount of other disks and the storage capacity of other disks.
[0135] As an optional implementation of the embodiment of the present application, the calculation module 503 is also used to: when the expected storage capacity of other disks is less than the storage capacity of the source disk, obtain the reserved storage capacity of other disks; calculate the maximum storage capacity of other disks based on the reserved storage capacity of other disks and the storage capacity of other disks; the migration module 504 is also used to migrate the block files in the task list to other disks in priority order when the maximum storage capacity of other disks is greater than the storage capacity of the source disk.
[0136] As an optional implementation of the embodiment of the present application, the calculation module 503 is also used to: when the maximum storage capacity of other disks is less than the storage capacity of the source disk, determine whether to migrate across nodes; the migration module 504 is also used to: if so, migrate the block files in the task list to other disks in order of priority until the storage capacity of other disks reaches the maximum storage capacity, and then migrate the remaining block files across nodes to the disks of other data nodes.
[0137] As an optional implementation of an embodiment of the present application, the preset replica status includes: a first replica status, indicating that the replica is in the Finalized status where writing has been completed and an additional write operation can be performed; a second replica status, indicating that the replica is in the Finalized status and no additional write operations are performed; a third replica status, indicating that the replica is in the writing status.
[0138] As an optional implementation of the embodiment of the present application, the migration module 504 is specifically used to: when the expected storage capacity of other disks is greater than the storage capacity of the source disk, migrate the block files corresponding to the first copy state to other disks, and then migrate the block files corresponding to the second copy state to other disks; for the block files corresponding to the third copy state, after a preset waiting time, migrate the target block files to other disks; wherein the target block files are block files converted to the Finalized state among the block files corresponding to the third copy state.
[0139] For the specific definition of the data migration device based on the HDFS block state, please refer to the definition of the data migration method based on the HDFS block state in the above text, which will not be repeated here. Each module in the above-mentioned data migration device based on the HDFS block state can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0140] In one embodiment, the present application provides an electronic device, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, an operator network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for detecting a jam is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball or a touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.
[0141] Those skilled in the art will understand that Figure 6 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0142] In one embodiment, the data migration device based on HDFS block status provided by the present application can be implemented in the form of a computer program. Figure 6 The electronic device shown in the figure is run. The memory of the electronic device can store various program modules constituting the data migration device based on HDFS block status. The computer program composed of various program modules enables the processor to execute the steps of the data migration method based on HDFS block status in various embodiments of the present application described in this specification.
[0143] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed, the steps of the data migration method based on HDFS block status of each embodiment of the present application described in this specification are implemented.
[0144] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.
[0145] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0146] In the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0147] In this application, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0148] In this application, computer-readable media includes permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0149] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0150] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / datanode>
Claims
1. A data migration method based on HDFS block status, characterized in that: include: Set the source disk to enter maintenance mode; Counting the copy status of the block files in the source disk and generating a task list; The task list includes block files corresponding to preset replica states; Obtaining the storage amount and capacity of the source disk and other disks, and calculating the expected storage amount of other disks; the source disk and the other disks belong to the same data node; In the case that the expected storage capacity of the other disk is greater than the storage capacity of the source disk, the block files in the task list are migrated to the other disk in a priority order, and the priority order corresponds to the preset copy state.
2. The method according to claim 1, characterized in that The obtaining of the storage amount and capacity of the source disk and other disks and calculating the expected storage amount of other disks includes: Calculate the total storage capacity of the data node according to the storage capacity of the source disk and other disks; Calculate the expected total capacity of the data node based on the capacity of other disks; Calculate the expected storage rate according to the total storage amount and the expected total capacity of the data node; Calculating the expected storage capacity of the other disks according to the expected storage rate and the capacity of the other disks; The expected storage capacity of the other disk is calculated according to the expected storage capacity of the other disk and the storage capacity of the other disk.
3. The method according to claim 1 or 2, characterized in that: After obtaining the storage amount and capacity of the source disk and other disks and calculating the expected storage amount of other disks, the method further includes: When the expected storage capacity of the other disk is less than the storage capacity of the source disk, obtaining the reserved storage capacity of the other disk; Calculate the maximum storage capacity of the other disks according to the reserved storage capacity of the other disks and the storage capacity of the other disks; When the maximum storage capacity of other disks is greater than the storage capacity of the source disk, the block files in the task list are migrated to the other disks in order of priority.
4. The method according to claim 3, characterized in that After calculating the maximum storage capacity of the other disks according to the reserved storage capacity of the other disks and the storage capacity of the other disks, the method further includes: When the maximum storage capacity of other disks is less than the storage capacity of the source disk, determining whether to enable cross-node migration; If so, the block files in the task list are migrated to the other disks in order of priority until the storage capacity of the other disks reaches the maximum storage capacity, and then the remaining block files are migrated across nodes to the disks of other data nodes.
5. The method according to claim 1, characterized in that The preset copy status includes: The first replica state indicates that the replica is in the Finalized state and can perform additional write operations; The second replica status indicates that it is in the Finalized state and no additional write operations are performed; The third replica state indicates that the replica is being written.
6. The method according to claim 5, characterized in that When the expected storage capacity of the other disk is greater than the storage capacity of the source disk, migrating the block files in the task list to the other disk in order of priority includes: When the expected storage capacity of the other disk is greater than the storage capacity of the source disk, the block files corresponding to the first copy status are migrated to the other disk, and then the block files corresponding to the second copy status are migrated to the other disk; For the block file corresponding to the third replica state, after a preset waiting time, the target block file is migrated to the other disk; wherein the target block file is a block file converted to the Finalized state among the block files corresponding to the third replica state.
7. A data migration device based on HDFS block status, characterized in that: include: The setting module is used to set the source disk to enter the maintenance mode; A statistics module, used to count the copy status of the block files in the source disk and generate a task list; The task list includes block files corresponding to the preset replica states; A calculation module, used to obtain the storage amount and capacity of the source disk and other disks, and calculate the expected storage amount of other disks; the source disk and the other disks belong to the same data node; A migration module is used to migrate the block files in the task list to the other disks in a priority order when the expected storage capacity of the other disks is greater than the storage capacity of the source disk, and the priority order corresponds to the preset copy state.
8. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the data migration method based on the HDFS block state as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the data migration method based on the HDFS block state according to any one of claims 1 to 6 is implemented.
10. A computer program product, characterized in that include: The computer program product comprises a computer program, and when the computer program is executed on a computer, the computer is enabled to implement the data migration method based on the HDFS block state as claimed in any one of claims 1 to 6.
Citation Information
Patent Citations
Data migration method and system for distributed file system and related components
CN111078120A
Self-adaptive time sequence data management system and method of time sequence database
CN115422152A
Disk replacement method and server
CN118331783A
Data storage method and device, electronic equipment and computer program product
CN118444849A
Guest-assisted live storage migration
US20240184605A1
