Data migration method and device based on HDFS (Hadoop Distributed File System), electronic equipment and storage medium

Abstract

The invention relates to a data migration method and device based on an HDFS (Hadoop Distributed File System), electronic equipment and a storage medium, and particularly relates to the technical field of data processing. Comprising the steps of obtaining the capacity and usage amount of a disk on a data node; the disk comprises a source disk to be unloaded and other disks; setting the volume data density of the source disk as a preset value, and calculating the volume data density of other disks; and according to the volume data density of the other disks, migrating the data of the source disk to the other disks, and balancing the data among the other disks, so as to maintain the data balance state of the data node after the source disk is unloaded. According to the embodiment of the invention, data of the source disk can be completely cleaned, and safe unloading of the source disk is supported; and the data is balanced among other disks through the volume data density, so that the data on each disk on the data node is uniformly distributed, and the stability of data reading and writing after the source disk is unloaded is favorably maintained.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data migration method, device, electronic device and storage medium based on HDFS. Background Art

[0002] HDFS is a distributed file system that usually stores data on multiple disks on multiple data nodes. However, due to a large number of data write and delete operations and disk replacement, data imbalance may occur between multiple disks in a DataNode.

[0003] To solve this problem, the DiskBalancer tool was introduced in Hadoop 3.0. DiskBalancer focuses on data balance within DataNode. It can evenly distribute data among multiple disks under a DataNode. The DiskBalancer tool supports two main functions, reporting and balancing. It can achieve data balance within DataNode by moving data from one disk to other disks. This ensures that data is evenly distributed among the disks and improves system performance and throughput. However, the DiskBalancer tool cannot meet the needs of disk offline and disk replacement. It can only balance storage between disks and cannot clear all source disk data. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a data migration method, device, electronic device and storage medium based on HDFS, which can meet the needs of data node unloading or replacing disks, clean up source disk data, and ensure data balance among each disk within the data node.

[0005] In order to achieve the above purpose, the technical solutions provided by the embodiments of the present application are as follows:

[0006] In a first aspect, the present application provides a data migration method based on HDFS, including: obtaining the capacity and usage of the disk on the data node; the disk includes a source disk to be uninstalled and other disks; setting the volume data density of the source disk to a preset value, and calculating the volume data density of other disks; migrating the data of the source disk to other disks according to the volume data density of other disks, and balancing the data between other disks to maintain the data balance state of the data node after uninstalling the source disk.

[0007] As an optional implementation of the embodiment of the present application, the volume data density of other disks is calculated, including: calculating the usage rate of each disk based on the capacity and usage of the disk on the data node; calculating the expected total capacity of the data node based on the capacity of other disks; calculating the total usage of the data node based on the usage of the disk on the data node; calculating the ideal usage rate of other disks based on the expected total capacity and the total usage; calculating the volume data density of other disks based on the ideal usage rate of other disks and the usage rate of other disks.

[0008] As an optional implementation of the embodiment of the present application, data of the source disk is migrated to other disks and data between other disks is balanced according to the volume data density of other disks, including: determining a target disk from other disks according to the volume data density of other disks; migrating data of the source disk to the target disk, and balancing data between other disks.

[0009] As an optional implementation of the embodiment of the present application, before migrating the data of the source disk to other disks according to the volume data density of other disks and balancing the data between other disks, the method also includes: calculating the ideal usage of other disks according to the ideal usage of other disks and the capacity of other disks; calculating the migratable amount of other disks according to the ideal usage of other disks and the usage of other disks.

[0010] As an optional implementation of the embodiment of the present application, data of the source disk is migrated to other disks according to the volume data density of other disks, and data between other disks is balanced, including: when the volume data density of the target disk is greater than the expected value among the volume data density of other disks, the corresponding data of the source disk is migrated to the target disk according to the migratable amount of the target disk; after the data migration of the source disk is completed, if the volume data density of the first disk is less than the expected value, and the volume data density of the second disk is greater than the expected value, the corresponding data of the first disk is migrated to the second disk according to the migratable amount of the second disk to balance the data between the first disk and the second disk; wherein the first disk and the second disk are any two disks among the other disks.

[0011] As an optional implementation of the embodiment of the present application, data of the source disk is migrated to other disks according to the volume data density of other disks, and data between other disks is balanced, including: determining m target disks from other disks according to the volume data density of other disks; splitting the source disk data into m parts according to the migratable amount of the m target disks, and migrating them to the m target disks accordingly; for n disks among the other disks except the m target disks, migrating data according to the migratable amount corresponding to the n disks.

[0012] As an optional implementation of the embodiment of the present application, other disks include original disks on the data node except the source disk, and newly added disks pre-mounted to the data node.

[0013] In a second aspect, the present application provides a data migration device based on HDFS, the device comprising:

[0014] The acquisition module is used to obtain the capacity and usage of the disk on the data node; the disk includes the source disk to be uninstalled and other disks;

[0015] A calculation module, used to set the volume data density of the source disk to a preset value and calculate the volume data density of other disks;

[0016] The migration module is used to migrate data of the source disk to other disks according to the volume data density of the other disks, and balance the data between the other disks to maintain the data balance state of the data node after the source disk is uninstalled.

[0017] As an optional implementation of the embodiment of the present application, the calculation module is specifically used to: calculate the utilization rate of each disk based on the capacity and usage of the disk on the data node; calculate the expected total capacity of the data node based on the capacity of other disks; calculate the total usage of the data node based on the usage of the disk on the data node; calculate the ideal utilization rate of other disks based on the expected total capacity and the total usage; calculate the volume data density of other disks based on the ideal utilization rate of other disks and the utilization rate of other disks.

[0018] As an optional implementation of the embodiment of the present application, the migration module is specifically used to: determine the target disk from other disks based on the volume data density of other disks; migrate the data of the source disk to the target disk, and balance the data between other disks.

[0019] As an optional implementation of the embodiment of the present application, the migration module is also specifically used to: calculate the ideal usage of other disks based on the ideal usage rate of other disks and the capacity of other disks; calculate the migratable amount of other disks based on the ideal usage of other disks and the usage of other disks.

[0020] As an optional implementation of the embodiment of the present application, the migration module is specifically used to: when the volume data density of the target disk is greater than the expected value among the volume data density of other disks, migrate the corresponding source disk data to the target disk according to the migratable amount of the target disk; after the data migration of the source disk is completed, if the volume data density of the first disk is less than the expected value and the volume data density of the second disk is greater than the expected value, migrate the corresponding first disk data to the second disk according to the migratable amount of the second disk to balance the data between the first disk and the second disk; wherein the first disk and the second disk are any two disks among the other disks.

[0021] As an optional implementation of the embodiment of the present application, the migration module is specifically used to: determine m target disks from other disks according to the volume data density of other disks; split the source disk data into m parts according to the migratable amount of the m target disks, and migrate them to the m target disks accordingly; for the n disks among the other disks except the m target disks, migrate out the data according to the migratable amount corresponding to the n disks.

[0022] In a third aspect, the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the HDFS-based data migration method as described in the first aspect or any one of its optional embodiments is implemented.

[0023] In a fourth aspect, the present application provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the HDFS-based data migration method as described in the first aspect or any one of its optional embodiments.

[0024] In a fifth aspect, the present application provides a computer program product, including: the computer program product includes a computer program, and when the computer program runs on a computer, the computer implements the HDFS-based data migration method as described in the first aspect or any one of its optional embodiments.

[0025] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:

[0026] The disclosed embodiment provides a data migration method based on HDFS, firstly obtaining the capacity and usage of the source disk to be uninstalled and other disks on the data node, setting the volume data density of the source disk to a preset value, and calculating the volume data density of other disks; then, according to the volume data density of other disks, migrating the data of the source disk to other disks, and balancing the data between other disks, so that the data node maintains a data balance state after uninstalling the source disk. In this way, the present application specifically sets the volume data density of the source disk, and the volume data density of other disks is calculated by the capacity and usage of each disk, which can reflect the current utilization of other disks, thereby migrating the source disk data according to the volume data density of other disks, so that the source disk data can be completely cleaned up, and the source disk can be safely uninstalled; and the volume data density is also used to balance the data between other disks, so that the data on each disk on the data node is evenly distributed, which is conducive to maintaining the stability of data reading and writing after the source disk is uninstalled. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0029] Figure 1A This is a diagram of the directory structure of a data node on a local disk;

[0030] Figure 1B This is a pseudo code diagram of the HDFS replica status;

[0031] Figure 2A A schematic diagram of a scenario provided in an embodiment of the present application is shown in FIG. 1 ;

[0032] Figure 2B A second schematic diagram of a scenario provided for an embodiment of the present application;

[0033] Figure 3 A flowchart of a data migration method based on HDFS according to an embodiment of the present application;

[0034] Figure 4 A schematic diagram of the structure of a data migration device based on HDFS provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of the structure of an electronic device described in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the technical terms required to be used in the embodiments or the prior art description are briefly introduced below:

[0037] HDFS adopts a master-slave architecture, consisting of NameNode (NN) and DataNode. NameNode is the master node, which is mainly responsible for managing the HDFS namespace, parsing client requests, and controlling client access to HDFS. DataNode is a slave node, mainly used for physical storage, and its data volume depends on the size of the cluster.

[0038] NameNode is the master node of HDFS. Its main functions are: responsible for managing the HDFS namespace, cluster information and data blocks; maintaining the entire HDFS file directory tree, the metadata of the file directory and the list of data blocks corresponding to each file; accepting client operation requests; managing the mapping relationship between files and data blocks, and between data blocks and DataNodes.

[0039] DataNode is responsible for processing read and write requests from file system clients. It creates, copies, and deletes data blocks under the unified scheduling of NameNode. DataNode encapsulates a structure in memory: the data storage component (FsDataset), which is responsible for dynamically maintaining all data block (Block) sets on this node. Because Block files exist in various storage disks and directories, it is also responsible for maintaining the storage directory (i.e., disk). In DataNode, each storage directory is encapsulated into a volume (FsVolume), simulating the "mount point" under the Linux system. Under normal circumstances, FsVolume can be considered as an abstraction of the disk. Sometimes it is necessary to access the real disk (such as retrieving disk available space), so it is necessary for the abstraction to interact directly with the real disk, which is mainly implemented by calling respective Linux commands in Java.

[0040] In HDFS, Block, i.e. its replica data, is the basic storage unit of a file. Files are connected in an orderly and regular manner through Block file blocks. Replica data can be understood as the specific storage embodiment of Block. HDFS configures the size of Block through ${dfs.Blocksize}, which is 128MB by default. When a new Block is generated, the replica location determines the storage location of the replica in the cluster, which is determined by the location selection strategy and storage strategy.

[0041] Maintenance mode, Maintenance state also refers to a new state of data nodes in HDFS (Hadoop Distributed File System), which is designed to reduce the impact of cluster maintenance. In this state, the node will not trigger replication operations, will not accept read and write requests, and will not be selected as a storage node, allowing maintenance without affecting data availability. The basic purpose of this state is to allow nodes to temporarily leave the cluster without transferring the data on them, while ensuring the basic availability of the cluster and the accessibility of the data. This state is suitable for operations such as rolling upgrades and fast switching of services to deal with software and hardware failures.

[0042] The purpose of entering the Maintenance state is to ensure data availability, especially when the data node has the only copy of some data in the cluster. Compared with the Decommission process, the purpose of the Maintenance state is to reduce the impact on the cluster, because it focuses on ensuring the existence of at least one copy of the data, rather than the Decommission process, which may require the transfer of data to meet the copy coefficient requirements. Therefore, in the Maintenance state, unless there is no copy of the data on other nodes, basically no data transfer operation will occur, thereby reducing the impact of maintenance operations on the cluster. In general, the HDFS Maintenance state provides a mechanism that allows nodes to perform necessary maintenance work without seriously affecting the availability and integrity of cluster data, while ensuring the stability and reliability of the system.

[0043] If the DataNode is in the Maintenance state, HDFS's processing of the DataNode node has the following major impacts: The replication of block data in this node will not be triggered within the cluster. The DataNode will not receive read and write requests from the client. In the past, if the DataNode was in the Out of Service state, many exceptions would be thrown. The DataNode will not be selected as a storage node for block data. Balancer / Mover operations will also avoid moving data on this node. The DataNode in the Maintenance state is in a "maintenance" state, which is not equivalent to the Dead or Decommissioned state. It can actively let the NameNode know its status and can quickly switch back to the service state at any time.

[0044] The directory structure of the disk. The DataNode volumes correspond to the directory configured by dfs.datanode.data.dir, which is where the HDFSDataNode stores the blocks. It can be a comma-separated list of directories. For example Figure 1A As shown in the figure, the block pool directory (BlockPoolSlice), which starts with BP-xxxx, is the directory of a block pool (BlockPool), such as BP-880450491-10.0.0.8-1721802325640. Under the pool block directory, the current directory is the actual entry for this BlockPool to access data in this disk; scanner.cursor file: DataNode will regularly verify each blk file, and this file is used to record the location of the verification. The tmp directory is used to store some temporary operation data. Under the current directory, dfsUsed saves the disk usage statistics, which are refreshed every 10 minutes. The Finalized directory is used to store the Block files (such as blk_1073741836) and their validation files (such as blk_1073741836_1012.meta) that have been written. The validation file ends with meta, which is the metadata file of the Block, which saves the verification information, meta length, etc. The rbw directory stores the Block files being written, and will be moved to the Finalized directory after writing is completed; the VERSION file is used to record information related to Namespace and BlockPool. It should be noted that Figure 1A This is just an example of the directory structure of a local disk on a data node. The complete directory structure of a data node is shown in the following figure. Figure 1A Not shown in full.

[0045] The replica state is maintained by the DataNode node, which mainly displays the state and visibility of the current replica data. The state will always be there from the time the data arrives until it is deleted. This state also directly affects the Block state on the NameNode side. The replica state is defined in ReplicaState. The pseudo code of the replica state is as follows: Figure 1B shown.

[0046] The replica status includes: Replica Being Written (RBW), Replica is Finalized (Finalized), Replica Waiting to be Recovered (RWR), Replica is Under Recovery (RUR), and Temporary.

[0047] RBW, from the moment the DataNode creates a new replica locally and starts receiving Block data, the replica is in the RBW state. Data in this state will be temporarily stored in the current / BP-xxx / current / rbw directory. Once all the data of the Block is received, the data will be moved from the rbw directory to the Finalized directory, a process that will be handled by the DataNode itself. Data in this state will be quickly reported to the NameNode so that the NameNode can further process the Block based on the replica status.

[0048] Finalized: For a Block, at least one replica must be in the Finalized state to indicate that the Client has completed the update of the Block. In actual use, DataNode stores the most replica data in the Finalized state, so DataNode also uses more mechanisms to ensure efficient access and prevent such data from being lost. Finalized is implemented through FinalizedReplica. The replica in this state will save the corresponding checksum value, which is the value of blk_xxx.meta introduced earlier.

[0049] RWR, when DataNode goes down, the replica data being written will be different from the replicas on other nodes. When the service is restarted, part of the replica data in current / BP-xxxx / current / rbw needs to be restored to maintain consistency between different replicas of the same Block.

[0050] RUR, evolved from RWR, indicates that an existing replica is being recovered if in this state. Recovery means redefining the data content and generationStamp of the current replica to keep it consistent with the data of other replicas belonging to the same block. In the RWR and RUR states, the numBytes and generationStamp values ​​related to the replica are active and need to wait for the entire recovery process to be completed before they can be determined.

[0051] Temporary, the corresponding data directory is current / BP-xxxx / tmp. The replicas of this state are special, mainly storing data scheduled within the cluster. Usually there are two situations where this type of data is generated: (1) Balance operation within the cluster. When performing cluster-level load balancing, the target data node will temporarily place the received data in the tmp directory, and after all data is successfully received, it will be moved to the Finalized directory. (2) Replica data completion. When performing replica data completion on the Block, the received data will also be temporarily placed here, and after successful completion, it will be moved to the Finalized directory.

[0052] Block Pool Directory ( Figure 1A There are current and tmp directories under BP-880450491-10.0.0.8-1721802325640). The tmp directory is used to store copies of the Temporary state. The Finalized directory under the current directory stores copies of the Finalized state, and the rbw directory stores copies of the RBW, RWR, and RUR states.

[0053] It can be seen that all data files are stored in the Finalized directory and the rbw directory, and there will be no data files with the same Block ID on the same DataNode. Therefore, the data on one disk can be moved to another disk by migrating the blk file, and then the disk can be uninstalled to achieve the purpose of shrinking.

[0054] There are currently two solutions for reducing the storage capacity of an HDFS cluster: The first is to reduce the number of DataNode nodes. If there are a large number of DataNode nodes, and it is possible to reduce the CPU and memory resources while reducing the storage space, you can reduce several DataNode nodes and directly use the decommission provided by HDFS. This is the most common method. The reduction process involves the cross-node migration of a large amount of data, which will generate a large amount of intranet traffic and may affect the online load. Operation and maintenance personnel need to pay close attention and manually tune. If there are only three DataNode nodes left in the cluster, or the CPU or memory resources above cannot be reduced synchronously, this method cannot be used.

[0055] The second method is to reduce the number of disk directories on a single DataNode. That is, while keeping the number of DataNode nodes unchanged, reduce the disk space on each node. You can modify the dfs.datanode.data.dir parameter on the DataNode, delete one or more disk directories, and then wait for HDFS to automatically add copies. This method will also cause a large amount of data to move between nodes, generate a large amount of intranet traffic, and may affect the online load. Operation and maintenance personnel need to pay close attention and manually tune. In addition, if there are only two copies of the data, it will be relatively dangerous. Once a node has a problem or a disk fails when a disk directory is deleted, data loss is very likely to occur.

[0056] Disk replacement usually involves replacing a disk with a longer service life with a new disk with more stable performance, or replacing a disk with a smaller storage specification with a larger disk. During long-term operation, some hard disks may experience longer delays and slower read and write times due to the disk itself or frequent read and write operations. The "slow disk" phenomenon is common in distributed storage architectures: when a disk has a performance failure or the IO load is too high, causing the delay to increase, almost all user services will be seriously affected. The "slow disk phenomenon" will cause the performance of the entire cluster to degrade, and in severe cases, it may even cause data loss, service unavailability, and other problems. The slow disk phenomenon is exactly the same as the ancient barrel effect: the barrel effect refers to the maximum amount of water that a barrel can hold, which is not determined by the tallest plank, but by the shortest plank. Similarly, the performance of distributed storage is not determined by the fastest disk, but by the slowest disk.

[0057] Disk replacement method: Format and mount the hard disk to be added, modify dfs.datanode.data.dir in the configuration file hdfs-site.xml on the DataNode node, and set it to the directory to be used from now on (add a new mount point and delete the obsolete mount point). Configure the new disk to remove the old disk. Similar to the second solution of shrinking, waiting for HDFS to automatically supplement the replica will also cause a large amount of data to move between nodes.

[0058] DiskBalancer is a command-line tool that evenly distributes data across all disks of a DataNode. The disk balancer works by creating a plan and then executing that plan on the DataNode. A plan is a set of statements that describe how much data should be moved between two disks. A plan consists of multiple steps. The move plan includes the source disk, the destination disk, and the number of bytes to be moved. The disk balancer does not interfere with other processes because it limits how much data is copied per second. The disk balancer is not enabled by default on the cluster. To enable DiskBalancer, you must set dfs.disk.balancer.enabled to true in hdfs-site.xml. However, DiskBalancer cannot meet the needs of disk offline and disk replacement. The reasons are as follows: (1) DiskBalancer can only copy Finalized blocks, cannot process blocks in progress, and cannot stop disks from being selected by the disk selection strategy to perform new write operations. (2) It can only balance storage between disks and cannot completely clear the source disk data.

[0059] In order to solve some or all of the technical problems existing in the above-mentioned related technologies, the embodiment of the present application provides a data migration method, device, electronic device, and medium based on HDFS, wherein the method first obtains the capacity and usage of the source disk and other disks to be uninstalled on the data node, sets the volume data density of the source disk to a preset value, and calculates the volume data density of other disks; then, according to the volume data density of other disks, migrate the data of the source disk to other disks, and balance the data between other disks, so that the data node maintains a data balance state after uninstalling the source disk. In this way, the present application specifically sets the volume data density of the source disk, and the volume data density of other disks is calculated by the capacity and usage of each disk, which can reflect the current utilization of other disks, thereby migrating the source disk data according to the volume data density of other disks, so that the source disk data can be completely cleaned up, and the source disk can be safely uninstalled; and the volume data density is also used to balance the data between other disks, so that the data on each disk on the data node is evenly distributed, which is conducive to maintaining the stability of data reading and writing after the source disk is uninstalled.

[0060] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0061] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0062] A data migration method based on HDFS provided in the embodiments of the present application can be implemented by a data migration device or electronic device based on HDFS, and the electronic device includes but is not limited to a vehicle terminal, a server, a personal computer, a laptop computer, a tablet computer, a smart phone, etc. The operating system of the electronic device may include Android, a mobile operating system (iOS) developed by Apple, an operating system (Windows) developed by Microsoft Corporation of the United States, etc., and the embodiments of the present application are not limited to this. The electronic device can be run alone to implement the present application, or it can be connected to the network and implement the present application through interactive operations with other computer devices in the network. Among them, the network in which the electronic device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN) network, etc.

[0063] It should be noted that the protection scope of the HDFS-based data migration method described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0064] The method provided in the embodiment of the present application is applicable to disk shrinking and disk replacement scenarios. Disk shrinking can be uninstalling a disk and copying the data on it to other disks on the same data node. For example, Figure 2A As shown, Figure 2A The scenario shown includes four disks: Disk1, Disk2, Disk3, and Disk4. Disk1 is unmounted and 1 TB of data on it is copied to Disk2, Disk3, and Disk4.

[0065] Disk replacement is to first copy the data on a disk to a new disk on the same data node, and then uninstall the disk. For example, Figure 2A As shown, Figure 2A The scenario shown includes four disks: Disk1, Disk2, Disk3, and Disk4. Disk1 needs to be replaced with Disk5, and 1 TB of data on it needs to be copied to Disk2, Disk3, Disk4, and Disk5.

[0066] In the above scenario, the embodiment of the present application realizes the uninstallation and replacement of the disk, clears all the data on the disk and ensures that the data in the data node is evenly distributed and the data is evenly distributed among the disks.

[0067] like Figure 3 As shown, Figure 3The following is a flow chart of a data migration method based on HDFS according to an embodiment of the present application. The method can be executed by a data migration device based on HDFS, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 3 As shown, the method mainly includes the following steps S301 to S303:

[0068] S301. Obtain the capacity and usage of the disk on the data node.

[0069] The disks on the data node include source disks and other disks. The source disk is the disk to be uninstalled. It can be a disk that needs to be removed from the data node or a disk that is replaced by a new disk. You can configure dfs.datanode.data.dir to add the mount path of the new disk. For example, the data node includes 4 disks: Disk1, Disk2, Disk3, and Disk4. Assume that Disk1 is the source disk. Other disks include Disk2, Disk3, and Disk4.

[0070] Other disks are disks other than the source disk on the data node, including original disks other than the source disk, and newly added disks pre-mounted to the data node. Figure 2B As shown in the figure, in the original state, the data node includes 4 disks: Disk1, Disk2, Disk3 and Disk4. Assume that Disk1 is the source disk to be uninstalled and will be replaced by the newly added disk Disk5. Other disks include Disk2, Disk3, Disk4 and Disk5.

[0071] In some embodiments, before executing step S301, the data node is set to enter the maintenance state (Maintenance). In this way, the NameNode will skip the node when adding a new block file and will not add a new data block. During data migration, only the existing data block files are copied.

[0072] S302: Set the volume data density of the source disk to a preset value, and calculate the volume data density of other disks.

[0073] In this application, the volume data density (VolumeDataDensity) of the source disk is set to -1, indicating that the source disk will be uninstalled. The default value is -1.

[0074] In some embodiments, when executing step S302 and calculating the volume data density of other disks, the usage rate of each disk is calculated according to the capacity and usage of the disks on the data node; the expected total capacity of the data node is calculated according to the capacity of other disks; the total usage of the data node is calculated according to the usage of the disks on the data node; and the usage rate of the source disk is set to 0. For example, the ideal usage rate of Disk1 is set to 0.

[0075] Exemplary, reference Figure 2A In the disk scaling scenario shown, the capacity and usage of each disk on the data node are obtained through step S301: the capacity of Disk1 is 2TB, and the usage is 1TB; the capacity of Disk2 is 3TB, and the usage is 1TB; the capacity of Disk3 is 3TB, and the usage is 0.75TB; the capacity of Disk4 is 2TB, and the usage is 1.25TB. First, according to the capacity and usage of each disk, the usage rate of each disk is calculated: the usage rate of Disk1 is 0.5, the usage rate of Disk2 is 0.33, the usage rate of Disk3 is 0.25, and the usage rate of Disk4 is 0.625. Then, according to the capacity of other disks except the source disk (Disk1), the expected total capacity of the data node is calculated to be 8TB. ​​Then, according to the usage of each disk, the total usage of the data node is calculated to be 4TB. According to the total usage and expected total capacity of the data node, the ideal usage rate of other disks is calculated to be 0.5. Based on the ideal usage and the usage of other disks, the volume data density of other disks is calculated: the volume data density of Disk2 is 0.17, the volume data density of Disk3 is 0.25, and the volume data density of Disk4 is -0.125. As shown in Table 1, the ideal usage of the source disk Disk1 is set to 0, and the volume data density is set to -1.

[0076] Table 1

[0077] Disk Number Disk1 Disk2 Disk3 Disk4 capacity 2TB 3TB 3TB 2TB Usage 1TB 1TB 0.75TB 1.25TB Utilization 0.5 0.33 0.25 0.625 Ideal utilization rate 0 0.5 0.5 0.5 Volume Data Density -1 0.17 0.25 -0.125

[0078] For example, reference Figure 2BIn the disk replacement scenario shown, the capacity and usage of each disk on the data node are obtained through step S301, including the newly added disk Disk5. The capacity of Disk1 is 2TB, and the usage is 1TB; the capacity of Disk2 is 3TB, and the usage is 1TB; the capacity of Disk3 is 3TB, and the usage is 0.75TB; the capacity of Disk4 is 2TB, and the usage is 1.25TB; the capacity of Disk4 is 2TB, and the usage is 0TB. First, according to the capacity and usage of each disk, the usage rate of each disk is calculated: the usage rate of Disk1 is 0.5, the usage rate of Disk2 is 0.33, the usage rate of Disk3 is 0.25, the usage rate of Disk4 is 0.625, and the usage rate of Disk5 is 0. Then, according to the capacity of other disks except the source disk (Disk1), the expected total capacity of the data node is calculated to be 10TB. Then, according to the usage of each disk, the total usage of the data node is calculated to be 4TB. According to the total usage and expected total capacity of the data node, the ideal usage rate of other disks is calculated to be 0.4. Based on the ideal usage and the usage of other disks, the volume data density of other disks is calculated: the volume data density of Disk2 is 0.17, the volume data density of Disk3 is 0.25, the volume data density of Disk4 is -0.125, and the volume data density of Disk5 is 0.4. As shown in Table 2, the ideal usage of the source disk Disk1 is set to 0, and the volume data density is set to -1.

[0079] Table 2

[0080] Disk Number Disk1 Disk2 Disk3 Disk4 Disk5 capacity 2TB 3TB 3TB 2TB 2TB Usage 1TB 1TB 0.75TB 1.25TB 0TB Utilization 0.5 0.33 0.25 0.625 0 Ideal storage utilization 0 0.4 0.4 0.4 0.4 Volume Data Density -1 0.07 0.15 -0.225 0.4

[0081] A positive value for volume data density indicates that the disk is underutilized, while a negative value indicates that the disk is overutilized relative to the current ideal storage goal. A larger positive value indicates more free disk space and lower disk utilization; a larger negative value indicates less free disk space.

[0082] S303 . Migrate data from the source disk to other disks according to volume data density of other disks, and balance data between other disks to maintain a data balance state of the data node after uninstalling the source disk.

[0083] According to the volume data density of other disks, determine whether it is positive or negative, migrate the data of the source disk to the disk with positive volume data density, and migrate the extra data on the disk with negative volume data density to the disk with positive volume data density, so that the data of each disk on the data node is balanced, ensuring that the performance of the data node will not be affected after the source disk is uninstalled.

[0084] In some embodiments, when executing step S303, a target disk is determined from other disks according to the volume data density of other disks, data of the source disk is migrated to the target disk, and data among other disks is balanced.

[0085] Optionally, based on the volume data density of other disks, determine the disk with the largest value (greater than 0), use it as the target disk, migrate all the data on the source disk to the target disk, and then use the balancing tool to balance the data distribution between the disks on the data node.

[0086] In some embodiments, before executing step S303, the ideal usage of other disks is calculated based on the ideal usage rate of other disks and the capacity of other disks; the migratable amount of other disks is calculated based on the ideal usage of other disks and the usage of other disks. Then, when executing step S303, if the volume data density of the target disk is greater than the expected value among the volume data densities of other disks, the data of the corresponding source disk is migrated to the target disk based on the migratable amount of the target disk; after the data migration of the source disk is completed, if the volume data density of the first disk is less than the expected value and the volume data density of the second disk is greater than the expected value, the data of the corresponding first disk is migrated to the second disk based on the migratable amount of the second disk to balance the data between the first disk and the second disk; wherein the first disk and the second disk are any two disks among the other disks.

[0087] Among them, the volume data density of the target disk is greater than the expected value, and the expected value can be 0. When the volume data density of the disk is the expected value, it means that the current storage state is ideal, and there is no need to migrate data or write data. The volume data density of the target disk is greater than the expected value, which means that the target disk is not fully utilized and has a low utilization rate. Based on the calculated migratable amount of the target disk, an equal amount of source disk data is migrated to the target disk to achieve the migration of data on the source disk to be uninstalled, and the utilization rate of the target disk is improved after the migration. It can be understood that the number of target disks is greater than or equal to 1, which means that there may be multiple target disks on the data node that can carry the data on the source disk. For each target disk, a corresponding amount of source disk data is migrated to it until all the source disk data is migrated.

[0088] After the data migration of the source disk is completed, pay attention to the disks whose volume data density is less than the expected value (0). A volume data density less than 0 indicates that the data stored on the disk is excessive, and the calculated disk migration amount is a negative value, indicating that this part of the data can be migrated out of the disk. Then, the data corresponding to the migration amount is migrated to another disk whose volume data density is greater than the expected value, thereby balancing the data of each disk on the data node, so that the volume data density of all disks is updated to 0, the disk data is evenly distributed, and each disk is fully utilized.

[0089] Exemplarily, according to the ideal usage rate of other disks and the capacity of other disks, the ideal usage of other disks is calculated: the ideal usage of Disk2 is 1.5TB, the ideal usage of Disk3 is 1.5TB, and the ideal usage of Disk4 is 1TB. According to the usage and ideal usage of other disks, the migratable amount of other disks is calculated: the migratable amount of Disk2 is 0.5TB, the migratable amount of Disk3 is 0.75TB, and the migratable amount of Disk4 is -0.25TB. According to the migratable amount of other disks, the data of the source disk is migrated to other disks, 0.5TB of data in Disk1 is migrated to Disk2, and 0.5TB of data in Disk1 is migrated to Disk3. And the data between other disks is balanced, and 0.25TB of data in Disk4 is migrated to Disk3, so that the data balance state of the data node is maintained after the source disk is uninstalled. As shown in Table 3.

[0090] Table 3

[0091] Disk Number Disk1 Disk2 Disk3 Disk4 capacity 2TB 3TB 3TB 2TB Usage 1TB 1TB 0.75TB 1.25TB Utilization 0.5 0.33 0.25 0.625 Ideal utilization rate 0 0.5 0.5 0.5 Volume Data Density -1 0.17 0.25 -0.125 Ideal usage 0 1.5 1.5 1TB Migration amount 0 0.5TB 0.75TB -0.25TB

[0092] As another example, in a disk replacement scenario, a new disk Disk5 is added. According to the ideal usage rates of other disks and the capacities of other disks, the ideal usage of other disks is calculated: the ideal usage of Disk2 is 1.2TB, the ideal usage of Disk3 is 1.2TB, the ideal usage of Disk4 is 0.8TB, and the ideal usage of Disk2 is 0.8TB. According to the usage and ideal usage of other disks, the migratable amount of other disks is calculated: the migratable amount of Disk2 is 0.2TB, the migratable amount of Disk3 is 0.45TB, the migratable amount of Disk4 is -0.45TB, and the migratable amount of Disk5 is 0.8TB. According to the migratable amount of other disks, the data of the source disk is migrated to other disks, 0.2TB of data in Disk1 is migrated to Disk2, and 0.8TB of data in Disk1 is migrated to Disk5. And the data between other disks is balanced, and 0.45TB of data in Disk4 is migrated to Disk3, so that the data balance state of the data node is maintained after the source disk is uninstalled. As shown in Table 4.

[0093] Table 4

[0094]

[0095] Optionally, when executing step S303, based on the volume data density of other disks, the disk with the largest absolute value is determined, and it is judged whether its migratable amount is a positive value. If so, the data on the source disk is migrated to the disk according to the migratable amount; if not, the data corresponding to the migratable amount on the disk is migrated to the disk with a positive migratable amount.

[0096] In some embodiments, when executing step S303, m target disks are determined from the other disks based on the volume data density of the other disks; the source disk data is split into m parts and migrated to the m target disks accordingly; for the n disks among the other disks except the m target disks, the data is migrated out according to the migratable amount corresponding to the n disks.

[0097] Specifically, when executing step S303, m disks with a volume data density greater than the expected value are selected based on the volume data density of other disks, and then the data on the source disk is split into m parts and migrated to these m disks respectively. Optionally, regarding the splitting of the source disk data, the splitting can be performed based on the ratio of the migratable amounts of these m disks, so that the data on the source disk can be evenly migrated and evenly distributed after the migration. After the data migration on the source disk is completed, for the remaining n disks of other disks, their volume data density is less than the expected value, indicating that the data on them exceeds the amount, then according to their respective migratable amounts, the corresponding share of data is migrated to the disk with a volume data density greater than the expected value, so that the volume data density of each other disk is equal to the expected value.

[0098] In summary, the embodiment of the present application provides a data migration method based on HDFS, which first obtains the capacity and usage of the source disk to be uninstalled and other disks on the data node, sets the volume data density of the source disk to a preset value, and calculates the volume data density of other disks; then, according to the volume data density of other disks, migrates the data of the source disk to other disks, and balances the data between other disks, so that the data node maintains a data balance state after uninstalling the source disk. In this way, the present application specifically sets the volume data density of the source disk, and the volume data density of other disks is calculated by the capacity and usage of each disk, which can reflect the current utilization of other disks, thereby migrating the source disk data according to the volume data density of other disks, so that the source disk data can be completely cleaned up, and the source disk can be safely uninstalled; and the volume data density is also used to balance the data between other disks, so that the data on each disk on the data node is evenly distributed, which is conducive to maintaining the stability of data reading and writing after the source disk is uninstalled.

[0099] In some embodiments, the data migration method provided in the embodiments of the present application can be executed through a command line. After setting the data node to enter the maintenance mode, execute the following command:

[0100] Execute the Plan command: hdfs balancer-plan datanodename to generate an operation plan. As shown in Table 5, Table 5 shows the parameters, default values, and descriptions included in the Plan command. When uninstalling a disk, you need to specify the parameter mode as replace or replace_balance and specify the path information of the disk to be replaced.

[0101] Table 5

[0102]

[0103] Then execute the Execute command: hdfs balancer-execute<JSON file path> , migrate data.

[0104] Then execute the Query command hdfs balancer-query <datanode>, query the execution progress. After the data migration is completed, set the DataNode node to exit the maintenance mode and modify the dfs.datanode.data.dir configuration to remove the disk.

[0105] During the process, you can execute the Cancel command hdfs balancer-cancel<JSON file path> , cancel the operation plan.

[0106] The above embodiment completes data migration through operation instructions provided by the command line, and can also provide intuitive feedback on the execution progress, which is convenient for operation and maintenance personnel.

[0107] like Figure 4 As shown, Figure 4 A schematic diagram of a data migration device based on HDFS provided in an embodiment of the present application, the device comprising:

[0108] The acquisition module 401 is used to acquire the capacity and usage of the disk on the data node; the disk includes the source disk to be uninstalled and other disks;

[0109] A calculation module 402 is used to set the volume data density of the source disk to a preset value and calculate the volume data density of other disks;

[0110] The migration module 403 is used to migrate the data of the source disk to other disks according to the volume data density of other disks, and balance the data between other disks to maintain the data balance state of the data node after uninstalling the source disk.

[0111] As an optional implementation of the embodiment of the present application, the calculation module 402 is specifically used to: calculate the utilization rate of each disk based on the capacity and usage of the disk on the data node; calculate the expected total capacity of the data node based on the capacity of other disks; calculate the total usage of the data node based on the usage of the disk on the data node; calculate the ideal utilization rate of other disks based on the expected total capacity and the total usage; calculate the volume data density of other disks based on the ideal utilization rate of other disks and the utilization rate of other disks.

[0112] As an optional implementation of the embodiment of the present application, the migration module 403 is specifically used to: determine the target disk from other disks according to the volume data density of other disks; migrate the data of the source disk to the target disk, and balance the data between other disks.

[0113] As an optional implementation of the embodiment of the present application, the migration module 403 is also specifically used to: calculate the ideal usage of other disks based on the ideal usage rate of other disks and the capacity of other disks; calculate the migratable amount of other disks based on the ideal usage of other disks and the usage of other disks.

[0114] As an optional implementation of the embodiment of the present application, the migration module 403 is specifically used to: when the volume data density of the target disk is greater than the expected value among the volume data density of other disks, migrate the corresponding source disk data to the target disk according to the migratable amount of the target disk; after the data migration of the source disk is completed, if the volume data density of the first disk is less than the expected value and the volume data density of the second disk is greater than the expected value, migrate the corresponding first disk data to the second disk according to the migratable amount of the second disk to balance the data between the first disk and the second disk; wherein the first disk and the second disk are any two disks among the other disks.

[0115] As an optional implementation of the embodiment of the present application, the migration module 403 is specifically used to: determine m target disks from other disks according to the volume data density of other disks; split the source disk data into m parts according to the migratable amount of the m target disks, and migrate them to the m target disks accordingly; for the n disks among the other disks except the m target disks, migrate out the data according to the migratable amount corresponding to the n disks.

[0116] As an optional implementation of the embodiment of the present application, other disks include original disks on the data node except the source disk, and newly added disks pre-mounted to the data node.

[0117] For the specific definition of the data migration device based on HDFS, please refer to the definition of the data migration method based on HDFS above, which will not be repeated here. Each module in the above-mentioned data migration device based on HDFS can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0118] In one embodiment, the present application provides an electronic device, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, an operator network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for detecting a jam is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball or a touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0119] Those skilled in the art will understand that Figure 5 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0120] In one embodiment, the HDFS-based data migration device provided in the present application can be implemented in the form of a computer program. Figure 5 The electronic device shown in the figure is run. The memory of the electronic device can store various program modules constituting the data migration device based on HDFS. The computer program composed of various program modules enables the processor to execute the steps of the data migration method based on HDFS in various embodiments of the present application described in this specification.

[0121] In one embodiment, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by the computer program, implements the steps of the HDFS-based data migration method of each embodiment of the present application described in this specification.

[0122] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0123] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0124] In the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0125] In this application, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0126] In this application, computer-readable media includes permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0127] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0128] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / datanode>

Claims

1. A data migration method based on HDFS, characterized in that: include: Get the capacity and usage of the disk on the data node; the disk includes the source disk to be uninstalled and other disks; Setting the volume data density of the source disk to a preset value, and calculating the volume data density of the other disks; According to the volume data density of the other disks, the data of the source disk is migrated to the other disks, and the data between the other disks is balanced, so as to maintain the data balance state of the data node after the source disk is uninstalled.

2. The method according to claim 1, characterized in that The calculating the volume data density of the other disks includes: Calculate the usage rate of each disk based on the capacity and usage of the disk on the data node; Calculate the expected total capacity of the data node according to the capacity of the other disks; Calculate the total usage of the data node according to the usage of the disk on the data node; Calculate the ideal usage rate of the other disks according to the expected total capacity and the total usage; The volume data density of the other disks is calculated according to the ideal usage rate of the other disks and the usage rate of the other disks.

3. The method according to claim 1, characterized in that The step of migrating the data of the source disk to other disks according to the volume data density of the other disks, and balancing the data between the other disks, includes: Determining a target disk from the other disks according to the volume data density of the other disks; Migrate the data of the source disk to the target disk, and balance the data among other disks.

4. The method according to claim 2, characterized in that: Before migrating the data of the source disk to other disks according to the volume data density of the other disks and balancing the data between the other disks, the method further includes: Calculating the ideal usage of the other disks according to the ideal usage rates of the other disks and the capacities of the other disks; The migratable amount of the other disk is calculated according to the ideal usage amount of the other disk and the usage amount of the other disk.

5. The method according to claim 4, characterized in that The step of migrating the data of the source disk to other disks according to the volume data density of the other disks, and balancing the data between the other disks, includes: When the volume data density of the target disk among the volume data densities of the other disks is greater than an expected value, migrating the corresponding data of the source disk to the target disk according to the migratable amount of the target disk; After the data migration of the source disk is completed, if the volume data density of the first disk is less than the expected value, and the volume data density of the second disk is greater than the expected value, the corresponding data of the first disk is migrated to the second disk according to the migratable amount of the second disk, so as to balance the data between the first disk and the second disk; The first disk and the second disk are any two disks among the other disks.

6. The method according to claim 4, characterized in that The step of migrating the data of the source disk to other disks according to the volume data density of the other disks, and balancing the data between the other disks, includes: Determine m target disks from the other disks according to the volume data density of the other disks; According to the migratable amounts of the m target disks, the source disk data is split into m parts, and the parts are migrated to the m target disks accordingly; For n disks among the other disks except the m target disks, data is migrated out according to the migratable amounts corresponding to the n disks.

7. The method according to claim 1, characterized in that The other disks include original disks on the data node except the source disk, and newly added disks pre-mounted to the data node.

8. A data migration device based on HDFS, characterized in that: include: The acquisition module is used to obtain the capacity and usage of the disk on the data node; the disk includes the source disk to be uninstalled and other disks; A calculation module, used for setting the volume data density of the source disk to a preset value and calculating the volume data density of the other disks; The migration module is used to migrate the data of the source disk to other disks according to the volume data density of the other disks, and balance the data between the other disks to maintain the data balance state of the data node after uninstalling the source disk.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the HDFS-based data migration method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the HDFS-based data migration method according to any one of claims 1 to 7 is implemented.

Images