HDFS-based data migration method and device, electronic equipment and storage medium

Abstract

The application relates to an HDFS-based data migration method and device, electronic equipment and a storage medium, in particular to the technical field of data processing. The application comprises the following steps: acquiring the capacity and usage of disks on a data node; the disks comprise a source disk to be unloaded and other disks; setting the volume data density of the source disk to a preset value, calculating the volume data density of the other disks; according to the volume data density of the other disks, migrating the data of the source disk to the other disks, and balancing the data among the other disks, so as to maintain the data balance state of the data node after the source disk is unloaded. The embodiment of the application can clean up the data of the source disk completely, supports safe unloading of the source disk, and balances the data among the other disks through the volume data density, so that the data on each disk of the data node is uniformly distributed, and the stability of data reading and writing after the source disk is unloaded is maintained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a data migration method and device based on HDFS, electronic equipment and storage medium. BACKGROUND

[0002] HDFS is a distributed file system, which can usually store data on multiple disks on multiple data nodes (DataNode). However, due to a large number of data write and delete operations and disk replacement, etc., data imbalance may occur between multiple disks in the DataNode.

[0003] In order to solve this problem, the DiskBalancer tool is introduced in the Hadoop3.0 version. DiskBalancer focuses on the data balancing of DataNode, which can balance the data between multiple disks under one DataNode. DiskBalancer tool supports two main functions, namely reporting and balancing, which can achieve data balancing in DataNode by moving data from one disk to other disks. In this way, it can ensure that the data is evenly distributed among the disks, and improve the performance and throughput of the system. However, the DiskBalancer tool cannot meet the needs of disk offline and disk replacement. It can only achieve the balance of storage between disks, and cannot completely clean up the data of the source disk. SUMMARY

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a data migration method and device based on HDFS, electronic equipment and storage medium, which can meet the needs of data node disk unloading or replacement, clean up the data of the source disk, and ensure the data balancing of each disk in the data node.

[0005] In order to achieve the above purpose, the technical scheme provided by the embodiments of the present application is as follows:

[0006] In a first aspect, the present application provides a data migration method based on HDFS, comprising: acquiring the capacity and usage of the disks on the data node; the disks include a source disk to be unloaded and other disks; setting the volume data density of the source disk to a preset value, and calculating the volume data density of the other disks; according to the volume data density of the other disks, migrating the data of the source disk to the other disks, and balancing the data between the other disks, so as to maintain the data balancing state of the data node after unloading the source disk.

[0007] As an optional implementation of the embodiment of the present application, the method for calculating the volume data density of the other disks comprises: calculating the usage rate of each disk according to the capacity and usage of the disks on the data node; calculating the expected total capacity of the data node according to the capacity of the other disks; calculating the total usage of the data node according to the usage of the disks on the data node; calculating the ideal usage rate of the other disks according to the expected total capacity and the total usage; and calculating the volume data density of the other disks according to the ideal usage rate of the other disks and the usage of the other disks.

[0008] As an optional implementation of the embodiment of the present application, the method for migrating the data of the source disk to the other disks and balancing the data among the other disks according to the volume data density of the other disks comprises: determining the target disk from the other disks according to the volume data density of the other disks; and migrating the data of the source disk to the target disk and balancing the data among the other disks.

[0009] As an optional implementation of the embodiment of the present application, before the method for migrating the data of the source disk to the other disks and balancing the data among the other disks according to the volume data density of the other disks, the method further comprises: calculating the ideal total usage of the other disks according to the ideal usage rate of the other disks and the capacity of the other disks; and calculating the migratable amount of the other disks according to the ideal total usage of the other disks and the usage of the other disks.

[0010] As an optional implementation of the embodiment of the present application, the method for migrating the data of the source disk to the other disks and balancing the data among the other disks according to the volume data density of the other disks comprises: in the case that the volume data density of the target disk is greater than the expected value among the volume data density of the other disks, migrating the data of the corresponding source disk to the target disk according to the migratable amount of the target disk; and after the data of the source disk is migrated, in the case that the volume data density of the first disk is less than the expected value and the volume data density of the second disk is greater than the expected value, migrating the data of the corresponding first disk to the second disk according to the migratable amount of the second disk to balance the data between the first disk and the second disk; wherein the first disk and the second disk are any two disks among the other disks.

[0011] As an optional implementation of the embodiment of the present application, the method for migrating the data of the source disk to the other disks and balancing the data among the other disks according to the volume data density of the other disks comprises: determining m target disks from the other disks according to the volume data density of the other disks; splitting the data of the source disk into m parts according to the migratable amount of the m target disks and migrating the m parts to the m target disks correspondingly; and migrating the data of n disks other than the m target disks among the other disks according to the migratable amount of the n disks.

[0012] As an optional implementation of the embodiment of the present application, the other disks include original disks on the data node except the source disk, and the newly added disks pre-mounted to the data node.

[0013] In a second aspect, the present application provides a data migration device based on HDFS, which comprises:

[0014] The acquisition module is configured to acquire the capacity and the usage of the disks on the data node, wherein the disks include the source disk to be unloaded and other disks.

[0015] The calculation module is configured to set the volume data density of the source disk as a preset value, and calculate the volume data density of the other disks.

[0016] The migration module is configured to migrate the data of the source disk to the other disks according to the volume data density of the other disks, and balance the data among the other disks, so as to maintain the data balanced state of the data node after the source disk is unloaded.

[0017] As an optional implementation of the embodiment of the present application, the calculation module is specifically configured to calculate the usage rate of each disk according to the capacity and the usage of the disks on the data node, calculate the expected total capacity of the data node according to the capacity of the other disks, calculate the total usage of the data node according to the usage of the disks on the data node, calculate the ideal usage of the other disks according to the expected total capacity and the total usage, and calculate the volume data density of the other disks according to the ideal usage of the other disks and the usage rate of the other disks.

[0018] As an optional implementation of the embodiment of the present application, the migration module is specifically configured to determine a target disk from the other disks according to the volume data density of the other disks, migrate the data of the source disk to the target disk, and balance the data among the other disks.

[0019] As an optional implementation of the embodiment of the present application, the migration module is further specifically configured to calculate the ideal usage of the other disks according to the ideal usage of the other disks and the capacity of the other disks, and calculate the migratable amount of the other disks according to the ideal usage of the other disks and the usage of the other disks.

[0020] As an optional implementation of the embodiment of the present application, the migration module is specifically configured to: in the case that the volume data density of the target disk is greater than the expected value among the volume data densities of the other disks, migrating the data of the corresponding source disk to the target disk according to the migratable amount of the target disk; after the data of the source disk is migrated, in the case that the volume data density of the first disk is less than the expected value and the volume data density of the second disk is greater than the expected value, migrating the data of the corresponding first disk to the second disk according to the migratable amount of the second disk, so as to balance the data between the first disk and the second disk; wherein the first disk and the second disk are any two disks among the other disks.

[0021] As an optional implementation of the embodiment of the present application, the migration module is specifically configured to: determining m target disks from the other disks according to the volume data densities of the other disks; splitting the data of the source disk into m parts according to the migratable amounts of the m target disks, and migrating the m parts to the m target disks correspondingly; for n disks among the other disks except the m target disks, migrating the data out according to the migratable amounts of the n disks.

[0022] In a third aspect, the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, the HDFS-based data migration method according to the first aspect or any optional implementation thereof is implemented.

[0023] In a fourth aspect, the present application provides a computer readable storage medium, comprising: a computer program stored on the computer readable storage medium, and when the computer program is executed by a processor, the HDFS-based data migration method according to the first aspect or any optional implementation thereof is implemented.

[0024] In a fifth aspect, the present application provides a computer program product, comprising: the computer program product comprises a computer program, and when the computer program runs on a computer, the computer program makes the computer implement the HDFS-based data migration method according to the first aspect or any optional implementation thereof.

[0025] The technical scheme provided by the embodiment of the present application has the following advantages compared with the prior art:

[0026] The embodiment of the present disclosure provides a data migration method based on HDFS, first, the capacity and usage of a source disk and other disks to be unloaded on a data node are acquired, the volume data density of the source disk is set as a preset value, and the volume data density of the other disks is calculated; then, according to the volume data density of the other disks, the data of the source disk is migrated to the other disks, and the data among the other disks is balanced, so that the data node maintains a data balanced state after the source disk is unloaded. In this way, the volume data density of the source disk is specially set, and the volume data density of the other disks is calculated through the capacity and usage of each disk, which can reflect the current utilization of the other disks, so that the data of the source disk is migrated according to the volume data density of the other disks, the data of the source disk can be completely cleaned up, the source disk is supported to be safely unloaded, and the data among the other disks is also balanced through the volume data density, so that the data is evenly distributed on each disk of the data node, which is beneficial to maintaining the stability of data reading and writing after the source disk is unloaded. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced here. Obviously, for those skilled in the art, other drawings can also be obtained from these drawings without any creative labor.

[0029] Figure 1A A directory structure diagram of a data node on a local disk;

[0030] Figure 1B A pseudo code diagram of HDFS copy state;

[0031] Figure 2A A scene diagram one provided by the embodiment of the present application;

[0032] Figure 2B A scene diagram two provided by the embodiment of the present application;

[0033] Figure 3 A flow diagram of a data migration method based on HDFS provided by the embodiment of the present application;

[0034] Figure 4 A structure diagram of a data migration device based on HDFS provided by the embodiment of the present application;

[0035] Figure 5 A structure diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the technical terms needed to be used in the embodiments or the prior art description will be briefly introduced as follows:

[0037] HDFS adopts a master-slave architecture, which is composed of a NameNode (NN) and a DataNode. The NameNode is the master node, which is mainly responsible for managing the namespace of HDFS, parsing the requests of the client, and controlling the access of the client to HDFS. The DataNode is the slave node, which is mainly used as a physical storage, and the data volume thereof depends on the cluster size.

[0038] The NameNode is the master node of HDFS, and its main functions are as follows: responsible for managing the namespace of HDFS, cluster information and data blocks; maintaining the file directory tree of the entire HDFS, the meta information of the file directory and the data block list corresponding to each file; accepting the operation requests of the client; managing the mapping relationship between the file and the data block, and the data block and the DataNode.

[0039] The DataNode is responsible for processing the read-write requests of the file system client. Under the unified scheduling of the NameNode, the DataNode creates, replicates and deletes the data blocks. The DataNode encapsulates a structure of a data storage component (FsDataset) in the memory, which is responsible for dynamically maintaining all the data block (Block) sets on the node. Because the Block file exists in various storage disks and directories, the FsDataset is also responsible for maintaining the storage directory (i.e., the disk). In the DataNode, each storage directory is encapsulated into a volume (FsVolume), which simulates the "mounting point" in the Linux system. Under normal circumstances, the FsVolume can be considered as an abstraction of the disk. Sometimes, it is necessary to access the real disk (such as retrieving the available space of the disk), and thus it is necessary to abstract and directly interact with the real disk, which is mainly realized by calling the Linux commands through Java.

[0040] In HDFS, the Block is the basic storage unit of the file, and the file is connected in order and regularly through a Block file block. The replica data can be understood as the specific storage embodiment of the Block. HDFS configures the size of the Block block through ${dfs.Blocksize}, and the default size is 128 MB. When a new Block is generated, the replica position determines the storage position of the replica in the cluster, which is determined by the position selection strategy and the storage strategy.

[0041] Maintenance mode, Maintenance state is also a new state of data nodes in HDFS (Hadoop Distributed File System), which aims to reduce the impact of cluster maintenance. In this state, the node will not trigger replication operations, will not accept read-write requests, and will not be selected as a storage node, thereby allowing maintenance without affecting data availability. The basic purpose of this state is to allow the node to temporarily leave the cluster without transferring the data on it, while ensuring the basic availability of the cluster and the accessibility of the data. This state is suitable for operations such as rolling upgrades and quick service switching to deal with software and hardware failures.

[0042] The purpose of entering the Maintenance state is to ensure data availability, especially when the data node has the only copy of some data in the cluster. Compared with the Decommission process, the purpose of the Maintenance state is to reduce the impact on the cluster, as it mainly focuses on ensuring the existence of at least one data copy, rather than like the Decommission process, which may need to transfer data to meet the requirements of the replica factor. Therefore, in the Maintenance state, there is basically no data transfer operation unless there is no copy of the data on other nodes, thereby reducing the impact on the cluster caused by maintenance operations. In general, the HDFS Maintenance state provides a mechanism that allows nodes to perform necessary maintenance work without seriously affecting the data availability and integrity of the cluster, while ensuring the stability and reliability of the system.

[0043] If the DataNode is in the Maintenance state, HDFS has the following main effects on the node of the DataNode: replication of the block data in this node will not be triggered within the cluster. The DataNode will not receive read-write requests from the client. In the past, the DataNode will throw a lot of exceptions if it is in the Out of Service state. The DataNode will not be selected as a storage node for the block data. Balancer / Mover operations will also avoid moving data on this node. The DataNode in the Maintenance state is in a "maintenance" state, which is not equivalent to the Dead or Decommissioned state, and it can actively let the NameNode know its state and quickly switch back to the service state at any time.

[0044] The directory structure of the disk, the volumes of the DataNode correspond to the directories configured by dfs.datanode.data.dir, which is the place where the HDFS DataNode stores the Block. It can be a comma-separated list of directories. As shown in Figure 1A , the Block Pool directory (BlockPoolSlice) with the prefix BP-xxxx is the directory of a certain Block Pool (BlockPool), for example, BP-880450491-10.0.0.8-1721802325640. Under the pool block directory, the current directory is the entry for the BlockPool to access the data in this disk; the scanner.cursor file: the DataNode will periodically check each blk file, and this file is used to record which position is checked. The tmp directory is used to store some temporary operation data. Under the current directory, dfsUsed saves the disk usage statistics, which is refreshed every 10 minutes. The Finalized directory is used to store the Block file (such as blk_1073741836) and its verification file (such as blk_1073741836_1012.meta) that has been written. The verification file with the suffix meta is the metadata file of the Block, which saves the verification information, meta length, etc. The rbw directory stores the Block file being written, which will be moved to the Finalized directory after writing is completed; the VERSION file is used to record the information related to the Namespace and BlockPool. It should be noted that, Figure 1A , only the directory structure of the data node on a certain disk on the local is shown by way of example, and the complete directory structure of the data node Figure 1A is not shown in its entirety.

[0045] The replica state is maintained by the DataNode node where it is located, and mainly shows the state and visibility of the current replica data. From the arrival of the data to the deletion, the state will always accompany. This state also directly affects the Block state on the NameNode side. The replica state is defined in ReplicaState. The pseudo code of the replica state is shown in Figure 1B .

[0046] Replica Being Written (RBW), Replica is Finalized (Finalized), Replica Waiting to be Recovered (RWR), Replica is Under Recovery (RUR), Temporary.

[0047] RBW, the state of a replica is in RBW since the DataNode creates a new replica locally and starts to receive Block data. The data in this state is temporarily stored in the current / BP-xxx / current / rbw directory. Once all the Block data is received, the data is moved from the rbw directory to the Finalized directory, which is processed by the DataNode autonomously. The data in this state is quickly reported to the NameNode so that the NameNode can further process the Block according to the replica state.

[0048] Finalized, for a Block, at least one replica in the Finalized state is required to indicate that the Client's update of the Block is complete. In actual use, the DataNode stores the most replica data in the Finalized state, so the DataNode also adopts more mechanisms to ensure the efficiency of access, and such data is not lost. Finalized is implemented by FinalizedReplica, and the replica in this state saves the corresponding check value, that is, the value of blk_xxx.meta introduced earlier.

[0049] RWR, when the DataNode is down, the replica data being written will be different from the replicas on other nodes, and when the service is restarted, part of the replica data in current / BP-xxxx / current / rbw needs to be recovered to maintain consistency between different replicas of the same Block.

[0050] RUR, evolved from RWR, if in this state, it indicates that a recovery is being performed on an existing replica. Recovery refers to redefining the data content and generationStamp of the current replica to maintain consistency with other replicas under the same Block. In the RWR and RUR states, the numBytes and generationStamp values related to the replica are active and need to wait for the entire recovery process to be completed to determine them.

[0051] Temporary, the corresponding data directory is current / BP-xxxx / tmp. The copy of this state is special, and mainly stores the data scheduled inside the cluster. There are usually two cases that will produce such data: (1) Balance operation inside the cluster. When performing cluster-level load balancing, the target data node will temporarily put the received data into the tmp directory, and after all successful reception, it will be moved to the Finalized directory. (2) Replica data supplement. When performing replica data supplement on Block, the received data will also be temporarily placed here, and after successful supplement, it will be moved to the Finalized directory.

[0052] Block pool directory Figure 1A The current and tmp directories are under BP-880450491-10.0.0.8-1721802325640. The tmp directory is used to store the Temporary state copy. The Finalized directory under the current directory stores the Finalized state copy, and the rbw directory stores the RBW, RWR, and RUR state copies.

[0053] As can be seen, all data files exist in the Finalized directory and the rbw directory, and the same Block ID data files will not exist on the same DataNode. Therefore, by migrating blk files, the data on one disk can be moved to another disk, and then the disk can be unloaded to achieve the purpose of shrinking.

[0054] There are currently two solutions for HDFS cluster storage shrinkage: the first is to reduce the number of DataNode nodes. If the number of DataNode nodes is large and the storage space is allowed to be reduced while reducing CPU and memory resources, several DataNode nodes can be shrunk directly using the HDFS provided decommission. This is the most common method, and the shrinkage process involves a large amount of data cross-node migration, which will generate a large amount of internal network traffic and may affect the online load, and the operation and maintenance personnel need to keep close attention and manual tuning. If the cluster is left with only 3 DataNode nodes, or the CPU or memory resources on it cannot be reduced synchronously, this method cannot be used.

[0055] The second method is to reduce the disk directory on a single DataNode. That is, in the case of keeping the number of DataNode nodes unchanged, the disk space on each node is reduced, the dfs.datanode.data.dir parameter on the DataNode can be modified, one or more disk directories are deleted, and then the HDFS is automatically supplemented. This method will also cause a large amount of data movement between nodes, which will generate a large amount of internal network traffic and may affect the online load, and the operation and maintenance personnel need to keep close attention and manual tuning. In addition, if the data has only 2 copies, it is relatively dangerous. Once a disk directory is deleted, a node is out of order or a disk is damaged, it is extremely likely to cause data loss.

[0056] Disk replacement is usually replacing the disk with a long service life with a new disk with stable performance, or replacing the disk with a small storage specification with a larger specification. In the long-term running process, due to the disk itself or frequent reading and writing, part of the hard disk will appear the situation of large delay and slow reading and writing. Distributed storage architecture generally exists "slow disk" phenomenon: when a disk performance failure or IO load is too high, etc. cause the delay to become large, almost all user businesses will be seriously affected. The "slow disk phenomenon" will cause the performance of the entire cluster to decline, and in severe cases, it will even cause data loss, service unavailability and other problems. The slow disk phenomenon is similar to the ancient bucket effect: the maximum amount of water that a bucket can hold is not determined by the highest plank, but by the shortest plank. Similarly, the performance of distributed storage is not determined by the fastest disk, but by the slowest disk.

[0057] Disk replacement method: format and mount the hard disk to be added, modify the dfs.datanode.data.dir configuration file in the hdfs-site.xml configuration file on the DataNode node, and set it to the directory to be used from now on (add new mount points and delete obsolete mount points). Configure the new disk to drop the old disk, similar to the second shrinkage method, waiting for HDFS to automatically supplement the copy will also cause a large amount of data movement between nodes.

[0058] DiskBalancer is a command-line tool that can distribute data evenly across all disks of a DataNode. DiskBalancer works by creating a plan to run, and then executing the plan on the DataNode. A plan is a set of statements that describe how much data should be moved between two disks. A plan consists of multiple steps. A move plan includes the source disk, the target disk, and the number of bytes to move. DiskBalancer does not interfere with other processes because it limits how much data it copies per second. DiskBalancer is not enabled by default on a cluster. To enable DiskBalancer, you must set dfs.disk.balancer.enabled to true in hdfs-site.xml. But DiskBalancer cannot meet the needs of disk offline and disk replacement. The reasons are as follows: (1) DiskBalancer can only copy the finalized blocks, and cannot handle the blocks in writing, and cannot stop the disk from being selected by the disk strategy to perform new write operations. (2) Only the storage balance between disks can be achieved, and the source disk data cannot be completely cleaned up.

[0059] To solve the above problems in the related art, the embodiments of the present application provide a data migration method and device based on HDFS, an electronic device, and a medium. The method first acquires the capacity and usage of a source disk to be unloaded and other disks on a data node, sets the volume data density of the source disk to a preset value, and calculates the volume data density of the other disks. Then, the data of the source disk is migrated to the other disks according to the volume data density of the other disks, and the data between the other disks is balanced, so that the data node maintains a balanced state after the source disk is unloaded. In this way, the volume data density of the source disk is specially set, and the volume data density of the other disks is calculated based on the capacity and usage of the disks, which can reflect the current utilization of the other disks, so that the data of the source disk is migrated according to the volume data density of the other disks, the data of the source disk can be completely cleaned up, and the safe unloading of the source disk is supported. In addition, the data between the other disks is also balanced based on the volume data density, so that the data is evenly distributed on the disks of the data node, which is conducive to maintaining the stability of data reading and writing after the source disk is unloaded.

[0060] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the schemes of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0061] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of the present application, not all embodiments.

[0062] The HDFS-based data migration method provided in the embodiments of the present application can be implemented by an HDFS-based data migration device or an electronic device, which includes but is not limited to a car terminal, a server, a personal computer, a notebook computer, a tablet computer, a smart phone, and the like. The operating system of the electronic device can include Android, iOS (a mobile operating system developed by Apple Inc.), Windows (an operating system developed by Microsoft Corporation of the United States), and the like, and the embodiments of the present application do not limit this. The electronic device can be operated alone to implement the present application, or can be connected to a network and interact with other computer devices in the network to implement the present application. The network in which the electronic device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), and the like.

[0063] It should be noted that the protection scope of the HDFS-based data migration method described in the embodiments of the present application is not limited to the execution order of the steps listed in the embodiments, and any scheme implemented by adding, replacing or replacing steps of the prior art according to the principles of the present application is included in the protection scope of the present application.

[0064] The method provided in the embodiments of the present application is applicable to disk capacity reduction and disk replacement scenarios. Disk capacity reduction can be to unload a certain disk and copy the data thereon to other disks of the same data node. For example, as shown in FIG. 1, Figure 2A as shown in FIG. 2, Figure 2A the scenario shown in FIG. 3 includes four disks: Disk1, Disk2, Disk3 and Disk4; Disk1 is unloaded, and 1 TB of data thereon is copied to Disk2, Disk3 and Disk4.

[0065] Disk replacement is to copy the data on a certain disk to an added disk of the same data node, and then unload the disk. For example, as shown in FIG. 1, Figure 2A as shown in FIG. 2, Figure 2A the scenario shown in FIG. 3 includes four disks: Disk1, Disk2, Disk3 and Disk4; Disk1 is unloaded, and 1 TB of data thereon is copied to Disk2, Disk3 and Disk4.

[0066] In the above scenarios, the embodiments of the present application achieve disk unloading and replacement, completely clean up the data of Disk, and ensure that the data distribution in the data node is balanced, and the data is evenly distributed among the disks.

[0067] as shown in FIG. 1, Figure 3 as shown in FIG. 2, Figure 3A flowchart of a data migration method based on HDFS is shown in the embodiment of the present application. The method can be executed by a data migration device based on HDFS. The device can be implemented by software and / or hardware and can be integrated in an electronic device. As shown in Figure 3 The method mainly includes the following steps S301-S303.

[0068] S301, obtaining the capacity and usage of the disks on the data node.

[0069] The disks on the data node include the source disk and other disks. The source disk is the disk to be unloaded, which can be the disk to be removed from the data node or the disk to be replaced by a new disk. The mounting path of the new disk can be configured by dfs.datanode.data.dir. For example, the data node includes four disks: Disk1, Disk2, Disk3 and Disk4. It is assumed that Disk1 is the source disk. The other disks include Disk2, Disk3 and Disk4.

[0070] The other disks are the disks on the data node other than the source disk, including the original disks other than the source disk and the new disks mounted to the data node in advance. For example, as shown in Figure 2B The data node includes four disks: Disk1, Disk2, Disk3 and Disk4 in the original state. It is assumed that Disk1 is the source disk to be unloaded and will be replaced by a new disk Disk5. The other disks include Disk2, Disk3, Disk4 and Disk5.

[0071] In some embodiments, before step S301 is executed, the data node is set to enter the maintenance state. In this way, the NameNode will not add data blocks to the node when adding block files. Only the existing data block files are copied during data migration.

[0072] S302, setting the volume data density of the source disk to a preset value and calculating the volume data density of the other disks.

[0073] In the present application, the volume data density of the source disk is set to -1, indicating that the source disk will be unloaded. The preset value is -1.

[0074] In some embodiments, when step S302 is executed to calculate the volume data density of the other disks, the usage rate of each disk is calculated according to the capacity and usage of the disks on the data node; the expected total capacity of the data node is calculated according to the capacity of the other disks; the total usage of the data node is calculated according to the usage of the disks on the data node; and the usage rate of the source disk is set to 0. For example, the ideal usage rate of Disk1 is set to 0.

[0075] For example, refer to Figure 2A In the disk scaling-down scenario shown, step S301 obtains the capacity and usage of each disk on the data node: Disk1 has a capacity of 2TB and a usage of 1TB; Disk2 has a capacity of 3TB and a usage of 1TB; Disk3 has a capacity of 3TB and a usage of 0.75TB; Disk4 has a capacity of 2TB and a usage of 1.25TB. First, based on the capacity and usage of each disk, the utilization rate of each disk is calculated: Disk1 has a utilization rate of 0.5, Disk2 has a utilization rate of 0.33, Disk3 has a utilization rate of 0.25, and Disk4 has a utilization rate of 0.625. Then, based on the capacity of the other disks excluding the source disk (Disk1), the expected total capacity of the data node is calculated to be 8TB. ​​Furthermore, based on the usage of each disk, the total usage of the data node is calculated to be 4TB. Based on the total usage and expected total capacity of the data node, the ideal utilization rate of the other disks is calculated to be 0.5. Based on the ideal utilization and the utilization of other disks, the volume data density of the other disks is calculated: the volume data density of Disk2 is 0.17, the volume data density of Disk3 is 0.25, and the volume data density of Disk4 is -0.125. As shown in Table 1, the ideal utilization of the source disk Disk1 is set to 0, and the volume data density is set to -1.

[0076] Table 1

[0077] Disk Number Disk1 Disk2 Disk3 Disk4 Capacity 2TB 3TB 3TB 2TB Usage 1TB 1TB 0.75TB 1.25TB Usage Rate 0.5 0.33 0.25 0.625 Ideal Usage Rate 0 0.5 0.5 0.5 Volume Data Density -1 0.17 0.25 -0.125

[0078] For example, see reference. Figure 2BIn the illustrated disk replacement scenario, the capacities and usage amounts of the disks on the data node are obtained by step S301, including the newly added disk Disk5. The capacity of Disk1 is 2TB, and the usage amount is 1TB; the capacity of Disk2 is 3TB, and the usage amount is 1TB; the capacity of Disk3 is 3TB, and the usage amount is 0.75TB; the capacity of Disk4 is 2TB, and the usage amount is 1.25TB; and the capacity of Disk5 is 2TB, and the usage amount is 0TB. First, the usage rates of the disks are calculated according to the capacities and usage amounts of the disks: the usage rate of Disk1 is 0.5, the usage rate of Disk2 is 0.33, the usage rate of Disk3 is 0.25, the usage rate of Disk4 is 0.625, and the usage rate of Disk5 is 0. Then, the expected total capacity of the data node is calculated according to the capacities of the disks other than the source disk (Disk1), which is 10TB. Further, the total usage amount of the data node is calculated according to the usage amounts of the disks, which is 4TB. According to the total usage amount and the expected total capacity of the data node, the ideal usage rate of the other disks is calculated to be 0.4. According to the ideal usage rate and the usage rates of the other disks, the volume data densities of the other disks are calculated: the volume data density of Disk2 is 0.17, the volume data density of Disk3 is 0.25, the volume data density of Disk4 is -0.125, and the volume data density of Disk5 is 0.4. As shown in Table 2, the ideal usage rate of the source disk Disk1 is set to 0, and the volume data density is set to -1.

[0079] Table 2

[0080] Disk Number Disk1 Disk2 Disk3 Disk4 Disk5 Capacity 2TB 3TB 3TB 2TB 2TB Usage 1TB 1TB 0.75TB 1.25TB 0TB Usage Rate 0.5 0.33 0.25 0.625 0 Ideal Storage Usage Rate 0 0.4 0.4 0.4 0.4 Volume Data Density -1 0.07 0.15 -0.225 0.4

[0081] A positive volume data density indicates that a disk is not fully utilized, while a negative volume data density indicates that the utilization rate of a disk is too high relative to the current ideal storage target. The larger the positive value is, the more available space the disk has, and the lower the utilization rate of the disk is; the larger the negative value is, the less available space the disk has.

[0082] S303, according to the volume data densities of the other disks, migrating the data of the source disk to the other disks and balancing the data among the other disks to maintain the data balancing state of the data node after the source disk is unloaded.

[0083] According to the volume data densities of the other disks, it is determined whether the value is positive or negative, the data of the source disk is migrated to the disk with a positive volume data density, and the extra data on the disk with a negative volume data density is migrated to the disk with a positive volume data density, so that the data on each disk of the data node is balanced, and the performance of the data node is not affected after the source disk is unloaded.

[0084] In some embodiments, when step S303 is performed, a target disk is determined from other disks according to the volume data density of the other disks, data of the source disk is migrated to the target disk, and data among the other disks is balanced.

[0085] Optionally, a disk with the largest value (greater than 0) is determined from the other disks according to the volume data density of the other disks, the disk is taken as a target disk, data of the source disk is migrated to the target disk, and then the data distribution among the disks on the data node is balanced by the balancing tool.

[0086] In some embodiments, before step S303 is performed, the ideal usage of the other disks is calculated according to the ideal usage rate of the other disks and the capacity of the other disks, and the migratable amount of the other disks is calculated according to the ideal usage of the other disks and the usage of the other disks. Then, when step S303 is performed, if the volume data density of the target disk in the volume data density of the other disks is greater than the expected value, data of the corresponding source disk is migrated to the target disk according to the migratable amount of the target disk; after the data of the source disk is migrated, if the volume data density of the first disk is less than the expected value and the volume data density of the second disk is greater than the expected value, data of the corresponding first disk is migrated to the second disk according to the migratable amount of the second disk to balance the data between the first disk and the second disk; wherein the first disk and the second disk are any two disks in the other disks.

[0087] The volume data density of the target disk is greater than the expected value, and the expected value can be 0. When the volume data density of the disk is the expected value, it indicates that the current storage state is ideal, and neither data needs to be migrated out nor data needs to be written in. The volume data density of the target disk being greater than the expected value indicates that the target disk is not fully utilized and the usage rate is low. Then, according to the calculated migratable amount of the target disk, an equal amount of data of the source disk is migrated to the target disk to achieve migration of the data on the source disk to be unloaded, and the utilization rate of the target disk is improved after the migration. It can be understood that the number of target disks is greater than or equal to 1, indicating that there can be multiple target disks on the data node that can carry data on the source disk. For each target disk, a corresponding amount of data of the source disk is migrated thereto until all data of the source disk is migrated.

[0088] After the data of the source disk is migrated, attention is paid to the disk with the volume data density less than the expected value (0). The volume data density less than 0 indicates that the data stored on the disk has exceeded the amount. The calculated migratable amount of the disk is negative, indicating that this part of data can be migrated out of the disk. Then, the corresponding data of the migratable amount is migrated to another disk with the volume data density greater than the expected value, so as to balance the data among the disks on the data node, update the volume data density of all disks to 0, and evenly distribute the disk data, so that each disk is fully utilized.

[0089] For example, the ideal usage of other disks is calculated according to the ideal usage rate of other disks and the capacity of other disks: the ideal usage of Disk2 is 1.5 TB, the ideal usage of Disk3 is 1.5 TB, and the ideal usage of Disk4 is 1 TB. The migratable amount of other disks is calculated according to the usage and the ideal usage of other disks: the migratable amount of Disk2 is 0.5 TB, the migratable amount of Disk3 is 0.75 TB, and the migratable amount of Disk4 is -0.25 TB. The data of the source disk is migrated to other disks according to the migratable amount of other disks: 0.5 TB of data in Disk1 is migrated to Disk2, and 0.5 TB of data in Disk1 is migrated to Disk3. The data among other disks is balanced: 0.25 TB of data in Disk4 is migrated to Disk3, so that the data balance state of the data node is maintained after the source disk is unloaded. As shown in Table 3.

[0090] Table 3

[0091] Disk Number Disk1 Disk2 Disk3 Disk4 Capacity 2TB 3TB 3TB 2TB Usage 1TB 1TB 0.75TB 1.25TB Usage Rate 0.5 0.33 0.25 0.625 Ideal Usage Rate 0 0.5 0.5 0.5 Volume Data Density -1 0.17 0.25 -0.125 Ideal Usage 0 1.5 1.5 1TB Migratable 0 0.5TB 0.75TB -0.25TB

[0092] For another example, in the disk replacement scenario, a new disk Disk5 is added. The ideal usage of other disks is calculated according to the ideal usage rate of other disks and the capacity of other disks: the ideal usage of Disk2 is 1.2 TB, the ideal usage of Disk3 is 1.2 TB, the ideal usage of Disk4 is 0.8 TB, and the ideal usage of Disk2 is 0.8 TB. The migratable amount of other disks is calculated according to the usage and the ideal usage of other disks: the migratable amount of Disk2 is 0.2 TB, the migratable amount of Disk3 is 0.45 TB, the migratable amount of Disk4 is -0.45 TB, and the migratable amount of Disk5 is 0.8 TB. The data of the source disk is migrated to other disks according to the migratable amount of other disks: 0.2 TB of data in Disk1 is migrated to Disk2, and 0.8 TB of data in Disk1 is migrated to Disk5. The data among other disks is balanced: 0.45 TB of data in Disk4 is migrated to Disk3, so that the data balance state of the data node is maintained after the source disk is unloaded. As shown in Table 4.

[0093] Table 4

[0094]

[0095] Optionally, when step S303 is performed, the disk with the largest absolute value is determined according to the volume data density of other disks, and it is determined whether the migratable amount of the disk is positive. If yes, the data on the source disk is migrated to the disk according to the migratable amount. If no, the data corresponding to the migratable amount on the disk is migrated to the disk with the positive migratable amount.

[0096] In some embodiments, when step S303 is performed, m target disks are determined from the other disks according to the volume data densities of the other disks; the source disk data is split into m parts corresponding to migration to the m target disks; and data is migrated out according to the migratable amounts of n disks other than the m target disks among the other disks.

[0097] Specifically, when step S303 is performed, m disks with volume data densities greater than an expected value are selected according to the volume data densities of the other disks, and the data on the source disk is then split into m parts and migrated to the m disks respectively. Optionally, the splitting of the data on the source disk can be performed according to the proportions of the migratable amounts of the m disks, so that the data on the source disk can be evenly migrated and evenly distributed after migration. After the migration of the data on the source disk is completed, n disks remaining among the other disks have volume data densities less than the expected value, indicating that the data thereon is excessive, and data corresponding to the respective shares is then migrated out to disks with volume data densities greater than the expected value according to the migratable amounts of the respective disks, so that the volume data density of each of the other disks is equal to the expected value.

[0098] In summary, the embodiments of the present application provide a data migration method based on HDFS. The method first acquires the capacities and usage amounts of a source disk to be unloaded and other disks on a data node, sets the volume data density of the source disk to a preset value, and calculates the volume data densities of the other disks. Then, the data on the source disk is migrated to the other disks according to the volume data densities of the other disks, and the data among the other disks is balanced, so that the data node remains in a data balanced state after the source disk is unloaded. In this way, the volume data density of the source disk is specially set, and the volume data densities of the other disks are calculated based on the capacities and usage amounts of the respective disks, so as to reflect the current utilization of the other disks, and the data on the source disk is migrated according to the volume data densities of the other disks, so that the data on the source disk can be completely cleaned up, supporting safe unloading of the source disk. Moreover, the data among the other disks is balanced based on the volume data densities, so that the data on the respective disks on the data node is evenly distributed, which is conducive to maintaining the stability of data reading and writing after the source disk is unloaded.

[0099] In some embodiments, the data migration method provided by the embodiments of the present application can be executed through a command line. After the data node is set to enter a maintenance mode, the following command is executed:

[0100] The Plan command is executed: hdfs balancer-plan datanodename, to generate a running plan. As shown in Table 5, the parameters, default values and explanations included in the Plan command are shown. When the disk is unloaded, the parameter mode is specified as replace or replace_balance, and the path information of the disk to be replaced is specified.

[0101] Table 5

[0102]

[0103] Execute the command again: hdfs balancer-execute <JSON file path> to migrate data.

[0104] Further execute the Query command hdfs balancer-query <datanode>Check the execution progress. After the data migration is complete, set the DataNode to exit maintenance mode and modify the dfs.datanode.data.dir configuration to remove the disk.

[0105] The Cancel command can be executed during the process: hdfs balancer-cancel<JSON file path> The operation plan has been cancelled.

[0106] The above embodiments complete data migration through operation instructions provided by the command line, and can also intuitively provide feedback on the execution progress, making it convenient for operation and maintenance personnel.

[0107] like Figure 4 As shown, Figure 4 A schematic diagram of a data migration device based on HDFS provided in this application embodiment, the device comprising:

[0108] The acquisition module 401 is used to acquire the capacity and usage of the disks on the data nodes; the disks include the source disk to be unloaded and other disks.

[0109] The calculation module 402 is used to set the volume data density of the source disk to a preset value and calculate the volume data density of other disks.

[0110] The migration module 403 is used to migrate data from the source disk to other disks based on the volume data density of other disks, and to balance the data among other disks in order to maintain the data balance of the data nodes after the source disk is unloaded.

[0111] As an optional implementation of this application, the calculation module 402 is specifically used for: calculating the utilization rate of each disk based on the capacity and usage of the disks on the data node; calculating the expected total capacity of the data node based on the capacity of other disks; calculating the total usage of the data node based on the usage of the disks on the data node; calculating the ideal utilization rate of other disks based on the expected total capacity and total usage; and calculating the volume data density of other disks based on the ideal utilization rate of other disks and the utilization rate of other disks.

[0112] As an optional implementation of this application, the migration module 403 is specifically used for: determining the target disk from other disks based on the volume data density of other disks; migrating the data from the source disk to the target disk; and balancing the data among other disks.

[0113] As an optional implementation of this application, the migration module 403 is further specifically used to: calculate the ideal usage of other disks based on the ideal utilization rate and capacity of other disks; and calculate the migrateable amount of other disks based on the ideal usage and usage of other disks.

[0114] As an optional implementation of the embodiment of the present application, the migration module 403 is specifically configured to: in the case that the volume data density of the target disk is greater than the expected value among the volume data densities of the other disks, migrate the data of the corresponding source disk to the target disk according to the migratable amount of the target disk; and after the data migration of the source disk is completed, in the case that the volume data density of the first disk is less than the expected value and the volume data density of the second disk is greater than the expected value, migrate the data of the corresponding first disk to the second disk according to the migratable amount of the second disk to balance the data between the first disk and the second disk; wherein the first disk and the second disk are any two disks among the other disks.

[0115] As an optional implementation of the embodiment of the present application, the migration module 403 is specifically configured to: determine m target disks from the other disks according to the volume data densities of the other disks; split the data of the source disk into m parts according to the migratable amounts of the m target disks, and migrate the m parts to the m target disks correspondingly; and migrate the data out of n disks other than the m target disks among the other disks according to the migratable amounts of the n disks.

[0116] As an optional implementation of the embodiment of the present application, the other disks include original disks other than the source disk on the data node, and newly added disks pre-mounted to the data node.

[0117] The specific limitations of the HDFS-based data migration device can be referred to the limitations of the HDFS-based data migration method in the above, which will not be repeated here. Each module in the above HDFS-based data migration device can be realized by software, hardware and combinations thereof in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0118] In one embodiment, the present application provides an electronic device, which can be a terminal, and its internal structure diagram can be as shown in Figure 5 As shown in the figure. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the electronic device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through WIFI, operator network, near field communication (NFC) or other technologies. The computer program is executed by the processor to implement a stutter detection method. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0119] Those skilled in the art can understand that, Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0120] In one embodiment, the HDFS-based data migration apparatus provided by the present application can be implemented in the form of a computer program, which can run on an electronic device as shown in the figure. The memory of the electronic device can store various program modules constituting the HDFS-based data migration apparatus. The computer program constituted by the various program modules enables the processor to execute the steps in the HDFS-based data migration method of each embodiment of the present application described in the specification. Figure 5

[0121] In one embodiment, the present application provides a computer readable storage medium having a computer program stored thereon, which, when executed by a computer program, implements the steps in the HDFS-based data migration method of each embodiment of the present application described in the specification.

[0122] Those skilled in the art should understand that the embodiments of the present application can be provided in the form of a method, a system or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media containing computer usable program code.

[0123] ​It should be understood that all the devices and methods disclosed in the embodiments of the present application can be implemented by other ways. The device embodiments described above are only schematic. For instance, the flowcharts and the block diagrams in the embodiments of the present application illustrate the possible implementation ways, functions and operation of the devices, methods and computer program products according to the embodiments of the present application. In this regard, each block in the flowcharts or the block diagrams can represent a module, a program segment or a part of code which comprises one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementation ways, the functions noted in the blocks can occur in a different order from that noted in the flowcharts. For example, two consecutive blocks can actually be executed in parallel or in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or the block diagrams, and the combination of blocks in the flowcharts and / or the block diagrams, can be implemented by a dedicated hardware-based system, or can be implemented by a combination of dedicated hardware and computer instructions.

[0124] In the present application, the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0125] In the present application, the memory can include a non-persistent memory in a computer readable medium, random access memory (RAM) and / or non-volatile memory, etc. in the form of read-only memory (ROM) or flash memory (flash RAM). The memory is an example of the computer readable medium.

[0126] In this application, computer readable medium includes permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, which can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable medium does not include transitory media such as modulated data signals and carriers.

[0127] It should be noted that, in this text, relational terms such as "first" and "second" and the like are used only to distinguish one entity or operation from another, and do not necessarily require or imply there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0128] The above is only the specific embodiment of the present application, which enables those skilled in the art to understand or implement the present application. Various modifications of these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / datanode>

Claims

1. A data migration method based on HDFS, characterized in that, include: Obtain the capacity and usage of the disks on the data node; the disks include the source disk to be unloaded and other disks, the other disks include the original disks on the data node other than the source disk, and newly added disks pre-mounted to the data node; The volume data density of the source disk is set to a preset value, and the volume data density of the other disks is calculated, wherein the volume data density of the source disk is used to indicate that the source disk will be unloaded; Based on the volume data density of the other disks, the data of the source disk is migrated to the other disks, and the data among the other disks is balanced, so as to maintain the data balance of the data node after the source disk is unloaded; The calculation of the volume data density of the other disks includes: Calculate the utilization rate of each disk based on the capacity and usage of the disks on the data nodes; Calculate the expected total capacity of the data node based on the capacity of the other disks; Calculate the total usage of the data nodes based on the disk usage on the data nodes; Calculate the ideal utilization rate of the other disks based on the expected total capacity and the total usage. Calculate the volume data density of the other disks based on their ideal utilization and utilization. The step of migrating data from the source disk to other disks based on the volume data density of the other disks, and balancing the data among the other disks, includes: If the volume data density of the target disk is greater than the expected value among the volume data densities of the other disks, the data of the corresponding source disk will be migrated to the target disk according to the migration capacity of the target disk. After the data migration of the source disk is completed, if the volume data density of the first disk is less than the expected value and the volume data density of the second disk is greater than the expected value, the corresponding data of the first disk will be migrated to the second disk according to the migration capacity of the second disk, so as to balance the data between the first disk and the second disk. The first disk and the second disk are any two of the other disks.

2. The method according to claim 1, characterized in that, The step of migrating data from the source disk to other disks based on the volume data density of the other disks, and balancing data among the other disks, includes: The target disk is determined from the other disks based on the volume data density of the other disks; The data from the source disk is migrated to the target disk, and the data is balanced among other disks.

3. The method according to claim 1, characterized in that, Before migrating data from the source disk to other disks based on the volume data density of the other disks, and before balancing the data among the other disks, the method further includes: Calculate the ideal usage of the other disks based on their ideal utilization rate and capacity. Calculate the migrateable amount of the other disks based on their ideal usage and actual usage.

4. The method according to claim 3, characterized in that, The step of migrating data from the source disk to other disks based on the volume data density of the other disks, and balancing data among the other disks, includes: Based on the volume data density of the other disks, determine m target disks from the other disks; Based on the migrated capacity of the m target disks, the source disk data is split into m parts and migrated to the m target disks accordingly; For the n disks other than the m target disks, data is migrated out according to the migration capacity corresponding to the n disks.

5. A data migration device based on HDFS, characterized in that, include: The acquisition module is used to acquire the capacity and usage of the disks on the data node; the disks include the source disk to be unloaded and other disks, the other disks include the original disks on the data node other than the source disk, and newly added disks pre-mounted to the data node; The calculation module is used to set the volume data density of the source disk to a preset value and calculate the volume data density of the other disks, wherein the volume data density of the source disk is used to indicate that the source disk will be unloaded; The migration module is used to migrate the data of the source disk to other disks according to the volume data density of the other disks, and to balance the data among the other disks, so as to maintain the data balance of the data node after the source disk is unloaded. The calculation of the volume data density of the other disks includes: Calculate the utilization rate of each disk based on the capacity and usage of the disks on the data nodes; Calculate the expected total capacity of the data node based on the capacity of the other disks; Calculate the total usage of the data nodes based on the disk usage on the data nodes; Calculate the ideal utilization rate of the other disks based on the expected total capacity and the total usage. Calculate the volume data density of the other disks based on their ideal utilization and utilization. The step of migrating data from the source disk to other disks based on the volume data density of the other disks, and balancing the data among the other disks, includes: If the volume data density of the target disk is greater than the expected value among the volume data densities of the other disks, the data of the corresponding source disk will be migrated to the target disk according to the migration capacity of the target disk. After the data migration of the source disk is completed, if the volume data density of the first disk is less than the expected value and the volume data density of the second disk is greater than the expected value, the corresponding data of the first disk will be migrated to the second disk according to the migration capacity of the second disk, so as to balance the data between the first disk and the second disk. The first disk and the second disk are any two of the other disks.

6. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the HDFS-based data migration method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, include: A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the HDFS-based data migration method as described in any one of claims 1 to 4.

Images