Data migration method, device, equipment, medium and product based on HDFS block status

Abstract

This application relates to a data migration method, apparatus, device, medium, and product based on HDFS block status, particularly in the field of data processing technology. It includes: setting the source disk into maintenance mode; statistically analyzing the replica status of block files on the source disk and generating a task list; the task list includes block files corresponding to preset replica statuses; obtaining the storage volume and capacity of the source disk and other disks, and calculating the expected storage capacity of the other disks; ensuring the source disk and other disks belong to the same data node; and migrating the block files in the task list to other disks according to priority order, where the priority order corresponds to the preset replica status, if the expected storage capacity of other disks is greater than the storage capacity of the source disk. This application's embodiments improve the operational capabilities of HDFS disks by calculating the expected storage capacity of other disks, assessing whether other disks can support the data on the source disk, and avoiding internal network traffic caused by missing replicas, thereby reducing the risk of impact on cluster services.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data migration method, apparatus, device, medium and product based on HDFS block state. Background Technology

[0002] The highly fault-tolerant Hadoop Distributed File System (HDFS) is a core project of Hadoop. As a distributed file system, it carries the storage of the entire Hadoop ecosystem, providing scalable, high-throughput, and highly reliable data storage services for upper-layer applications and users in the Hadoop ecosystem. It is suitable for distributed reading and writing of large-scale data, especially in scenarios with more reads than writes.

[0003] HDFS typically stores data across multiple disks on multiple DataNodes. HDFS provides two options for unloading disks to achieve scaling down or disk replacement: Option 1 directly uses HDFS's decommissioning mechanism to reduce the number of DataNodes; Option 2 modifies the `dfs.datanode.data.dir` configuration parameter on the DataNode, which specifies the HDFS data block storage location, by removing the disk directory to be unloaded from its value. However, both of these options result in a large amount of data moving between nodes, generating significant internal network traffic that impacts cluster operations, and carries a high risk of data loss. Summary of the Invention

[0004] To address or at least partially address the aforementioned technical problems, this application provides a data migration method, apparatus, device, medium, and product based on HDFS block state, which can avoid internal network traffic generated when replicas are missing in order to replenish replicas, thereby reducing the risk of impact on cluster services.

[0005] To achieve the above objectives, the technical solutions provided in this application are as follows:

[0006] Firstly, this application provides a data migration method based on HDFS block status, comprising: setting the source disk to enter maintenance mode; statistically analyzing the replica status of block files in the source disk and generating a task list; the task list includes block files corresponding to preset replica statuses; obtaining the storage volume and capacity of the source disk and other disks, and calculating the expected storage capacity of other disks; the source disk and other disks belonging to the same data node; and, if the expected storage capacity of other disks is greater than the storage volume of the source disk, migrating the block files in the task list to other disks in priority order, the priority order corresponding to the preset replica statuses.

[0007] As an optional implementation of this application, obtaining the storage amount and capacity of the source disk and other disks, and calculating the expected storable amount of the other disks, includes: calculating the total storage of the data node based on the storage amount of the source disk and other disks; calculating the expected total capacity of the data node based on the capacity of the other disks; calculating the expected storage rate based on the total storage of the data node and the expected total capacity; calculating the expected storage amount of the other disks based on the expected storage rate and the capacity of the other disks; and calculating the expected storable amount of the other disks based on the expected storage amount of the other disks and the storage amount of the other disks.

[0008] As an optional implementation of this application, after obtaining the storage amount and capacity of the source disk and other disks and calculating the expected storage capacity of the other disks, the method further includes: if the expected storage capacity of the other disks is less than the storage amount of the source disk, obtaining the reserved storage amount of the other disks; calculating the maximum storage capacity of the other disks based on the reserved storage amount of the other disks and the storage amount of the other disks; and if the maximum storage capacity of the other disks is greater than the storage capacity of the source disk, migrating the block files in the task list to the other disks in priority order.

[0009] As an optional implementation of this application, after calculating the maximum storable amount of other disks based on the reserved storage amount and storage amount of other disks, the method further includes: if the maximum storable amount of other disks is less than the storage amount of the source disk, determining whether to migrate across nodes; if so, migrating the block files in the task list to other disks in priority order until the storage amount of other disks reaches the maximum storable amount, and then migrating the remaining block files across nodes to the disks of other data nodes.

[0010] As an optional implementation of this application, the preset replica states include: a first replica state, indicating that the replica has been written to the Finalized state and can be appended; a second replica state, indicating that the replica is in the Finalized state and no further appending operations are being performed; and a third replica state, indicating that the replica is being written to.

[0011] As an optional implementation of this application, when the expected storage capacity of other disks is greater than the storage capacity of the source disk, the block files in the task list are migrated to other disks in priority order, including: when the expected storage capacity of other disks is greater than the storage capacity of the source disk, migrating the block files corresponding to the first replica state to other disks, and then migrating the block files corresponding to the second replica state to other disks; for the block files corresponding to the third replica state, after a preset waiting time, migrating the target block file to other disks; wherein, the target block file is the block file in the block files corresponding to the third replica state that has been converted to the Finalized state.

[0012] Secondly, this application provides a data migration apparatus based on HDFS block state, the apparatus comprising:

[0013] The settings module is used to configure the source disk to enter maintenance mode;

[0014] The statistics module is used to count the replica status of block files in the source disk and generate a task list; the task list includes block files corresponding to preset replica statuses.

[0015] The calculation module is used to obtain the storage volume and capacity of the source disk and other disks, and to calculate the expected storage capacity of the other disks; the source disk and other disks belong to the same data node;

[0016] The migration module is used to migrate block files in the task list to other disks in priority order when the expected storage capacity of other disks is greater than the storage capacity of the source disk. The priority order corresponds to the preset copy status.

[0017] As an optional implementation of this application, the calculation module is specifically used for: calculating the total storage of the data node based on the storage capacity of the source disk and other disks; calculating the expected total capacity of the data node based on the capacity of other disks; calculating the expected storage rate based on the total storage of the data node and the expected total capacity; calculating the expected storage capacity of other disks based on the expected storage rate and the capacity of other disks; and calculating the expected storable capacity of other disks based on the expected storage capacity of other disks and the storage capacity of other disks.

[0018] As an optional implementation of this application, the calculation module is further configured to: obtain the reserved storage capacity of other disks when the expected storage capacity of other disks is less than the storage capacity of the source disk; calculate the maximum storage capacity of other disks based on the reserved storage capacity of other disks and the storage capacity of other disks; the migration module is further configured to migrate the block files in the task list to other disks in priority order when the maximum storage capacity of other disks is greater than the storage capacity of the source disk.

[0019] As an optional implementation of this application, the calculation module is further configured to: determine whether to migrate across nodes when the maximum storage capacity of other disks is less than the storage capacity of the source disk; the migration module is further configured to: if so, migrate the block files in the task list to other disks in priority order until the storage capacity of other disks reaches the maximum storage capacity, and then migrate the remaining block files across nodes to the disks of other data nodes.

[0020] As an optional implementation of this application, the preset replica states include: a first replica state, indicating that the replica has been written to the Finalized state and can be appended; a second replica state, indicating that the replica is in the Finalized state and no further appending operations are being performed; and a third replica state, indicating that the replica is being written to.

[0021] As an optional implementation of this application, the migration module is specifically used for: migrating the block file corresponding to the first replica state to the other disk when the expected storage capacity of the other disk is greater than the storage capacity of the source disk, and then migrating the block file corresponding to the second replica state to the other disk; for the block file corresponding to the third replica state, after a preset waiting time, migrating the target block file to the other disk; wherein, the target block file is the block file in the block file corresponding to the third replica state that has been converted to the Finalized state.

[0022] Thirdly, this application provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the data migration method based on HDFS block state as described in the first aspect or any of its optional embodiments.

[0023] Fourthly, this application provides a computer-readable storage medium, comprising: storing a computer program on the computer-readable storage medium, wherein when the computer program is executed by a processor, it implements the data migration method based on HDFS block state as described in the first aspect or any of its optional embodiments.

[0024] Fifthly, this application provides a computer program product, comprising: the computer program product including a computer program that, when the computer program is run on a computer, causes the computer to implement the data migration method based on HDFS block state as described in the first aspect or any of its optional embodiments.

[0025] The technical solution provided in this application has the following advantages compared with the prior art:

[0026] This disclosure provides a data migration method based on HDFS block status. First, the source disk is set to maintenance mode. Then, the replica status of block files on the source disk is statistically analyzed, generating a task list consisting of block files corresponding to preset replica statuses. Next, the storage capacity and volume of the source disk and other disks on the data node are obtained, and the expected storage capacity of the other disks is calculated. Furthermore, if the expected storage capacity of the other disks is greater than the storage capacity of the source disk, the block files in the task list are migrated to the other disks according to their priority order corresponding to the preset replica status. Thus, this application sets a maintenance mode at the disk level, avoiding adverse effects on other operations of the data node. By calculating the expected storage capacity of other disks, it assesses whether other disks can carry the data from the source disk, improving the operation and maintenance capabilities of HDFS disks, realizing intra-node data migration, and avoiding internal network traffic generated to replenish replicas when replicas are missing, thereby reducing the risk of cluster service disruptions. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1A This is a schematic diagram of the directory structure of a data node on a local disk.

[0030] Figure 1B A pseudocode diagram illustrating the HDFS replica state;

[0031] Figure 2 This is a flowchart illustrating a data migration method based on HDFS block state according to an embodiment of this application;

[0032] Figure 3A A pseudocode diagram illustrating the setting of the source disk into maintenance mode provided in an embodiment of this application;

[0033] Figure 3B A schematic diagram of parameters for setting the source disk to enter maintenance mode, provided in an embodiment of this application;

[0034] Figure 3C A schematic diagram of the execution command for setting the source disk to enter maintenance mode, provided in an embodiment of this application;

[0035] Figure 4AA schematic diagram illustrating the parameters, default values, and descriptions included in the Plan command provided in the embodiments of this application;

[0036] Figure 4B A schematic diagram illustrating the parameters, default values, and descriptions included in the Execute command provided in the embodiments of this application;

[0037] Figure 4C A schematic diagram illustrating the parameters, default values, and descriptions included in the Query command provided in the embodiments of this application;

[0038] Figure 4D A pseudocode diagram illustrating the Cancel command provided in an embodiment of this application;

[0039] Figure 5 A schematic diagram of a data migration device based on HDFS block state provided in this application embodiment;

[0040] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the technical terms used in the description of the embodiments or the prior art will be briefly introduced below:

[0042] HDFS employs a master-slave architecture, consisting of a NameNode (NN) and DataNodes. The NameNode, as the master node, is primarily responsible for managing the HDFS namespace, resolving client requests, and controlling client access to HDFS. DataNodes, as slave nodes, are mainly used for physical storage, and their data volume depends on the cluster size.

[0043] The NameNode is the master node of HDFS. Its main functions are: to manage the HDFS namespace, cluster information, and data blocks; to maintain the entire HDFS file directory tree, the metadata of the file directories, and the list of data blocks corresponding to each file; to accept client operation requests; and to manage the mapping relationship between files and data blocks, and between data blocks and DataNodes.

[0044] DataNodes handle read and write requests from file system clients. Under the unified scheduling of the NameNode, they create, replicate, and delete data blocks. Each DataNode encapsulates a structure in memory: a data storage component (FsDataset), responsible for dynamically maintaining the set of all data blocks on its node. Because block files exist on various storage disks and directories, it also maintains storage directories (i.e., disks). Within a DataNode, each storage directory is encapsulated as a volume (FsVolume), simulating a "mount point" in a Linux system. Normally, an FsVolume can be considered an abstraction of the disk. Sometimes, access to the actual disk is required (e.g., retrieving available disk space), necessitating direct interaction between the abstraction and the actual disk, primarily achieved through Java calling respective Linux commands.

[0045] In HDFS, a block, or its replica data, is the basic storage unit that makes up a file. Files are connected in an ordered and structured manner using blocks. Replica data can be understood as the concrete storage representation of a block. HDFS configures the block size using `${dfs.blocksize}`, with a default value of 128MB. When a new block is created, the replica location determines its storage location within the cluster, determined by both the location selection strategy and the storage strategy.

[0046] Maintenance mode, or Maintenance state, refers to a new state of data nodes in HDFS (Hadoop Distributed File System) designed to reduce the impact of cluster maintenance. In this state, nodes do not trigger replication operations, do not accept read / write requests, and are not elected as storage nodes, thus allowing maintenance to be performed without affecting data availability. The basic purpose of this state is to allow nodes to temporarily leave the cluster without migrating their data, while ensuring basic cluster availability and data accessibility. This state is suitable for operations such as rolling upgrades and rapid service failover to address hardware and software failures.

[0047] The purpose of entering the Maintenance state is to ensure data availability, especially when data nodes possess unique replicas of certain data within the cluster. Compared to the Decommission process, the Maintenance state aims to minimize the impact on the cluster because it primarily focuses on ensuring the existence of at least one data replica, rather than potentially requiring data migration to meet replica count requirements, as the Decommission process might. Therefore, in the Maintenance state, data migration operations are virtually nonexistent unless no other node possesses a replica of the data, thus reducing the impact of maintenance operations on the cluster. In summary, the HDFS Maintenance state provides a mechanism that allows nodes to perform necessary maintenance without severely impacting cluster data availability and integrity, while ensuring system stability and reliability. It should be noted that... Figure 1A This is merely an example illustrating the directory structure of a data node on a local disk. The complete directory structure of a data node is not shown. Figure 1A It is not shown in its entirety.

[0048] The disk directory structure, where DataNode volumes correspond to the directories configured via `dfs.datanode.data.dir`, is where HDFS DataNodes store blocks. It can be a comma-separated list of directories (typically, each directory resides on a different disk). For example... Figure 1AAs shown, the BlockPoolSlice directory, starting with BP-xxxx, represents a specific BlockPool, such as BP-880450491-10.0.0.8-1721802325640. Within the BlockPool directory, the `current` directory is the actual entry point for this BlockPool to access data on this disk. The `scanner.cursor` file records the location of the verification process as performed periodically by the DataNode on each blk file. The `tmp` directory stores temporary operation data. Under the `current` directory, `dfsUsed` stores disk usage statistics, refreshed every 10 minutes. The `Finalized` directory stores completed Block files (e.g., blk_1073741836) and their verification files (e.g., blk_1073741836_1012.meta). The verification file, ending with `.meta`, is the Block's metadata file, storing verification information, meta length, etc. The rbw directory stores the Block files being written; once the writing is complete, they will be moved to the Finalized directory. The VERSION file is used to record information related to Namespaces and BlockPools.

[0049] The replica state is maintained by the DataNode node and primarily displays the current status and visibility of the replica data. This state persists from the arrival of the data until its deletion. This state also directly affects the Block state on the NameNode. The replica state is defined in `ReplicaState`. The pseudocode for the replica state is as follows: Figure 1B As shown.

[0050] Replica status includes: Replica Being Written (RBW), Replica is Finalized (Finalized), Replica Waiting to be Recovered (RWR), Replica is Under Recovery (RUR), and Temporary.

[0051] RBW (Replica in RBW) begins when a DataNode creates a new replica locally and starts receiving Block data. Data in this state is temporarily stored in the `current / BP-xxx / current / rbw` directory. Once all data for a Block has been received, the data is moved from the `rbw` directory to the `Finalized` directory; this process is handled autonomously by the DataNode. Data in this state is quickly reported to the NameNode so that the NameNode can further process the Block based on its replica status.

[0052] For a block to be considered complete, at least one replica must be in the Finalized state. In practice, DataNodes store the most Finalized replicas, therefore employing multiple mechanisms to ensure efficient access and prevent data loss. Finalized is implemented through FinalizedReplica, where replicas in this state store the corresponding checksum, which is the value of blk_xxx.meta mentioned earlier.

[0053] RWR (Replica Write) means that when a DataNode fails, the replica data being written will differ from the replicas on other nodes. When the service restarts, a portion of the replica data in current / BP-xxxx / current / rbw needs to be recovered to maintain consistency between different replicas of the same Block.

[0054] RUR, evolved from RWR, indicates that a recovery of an existing replica is underway. Recovery refers to redefining the data content and generationStamp of the current replica to ensure consistency with other replicas belonging to the same block. In both RWR and RUR states, the replica-related numBytes and generationStamp values ​​are active and will only be determined after the entire recovery process is complete.

[0055] Temporary, the corresponding data directory is current / BP-xxxx / tmp. This type of replica is special, mainly storing data scheduled within the cluster. There are usually two situations that generate this type of data: (1) Balance operation within the cluster. When performing cluster-level load balancing, the target data node temporarily puts the received data into the tmp directory, and after all data is successfully received, it will be moved into the Finalized directory. (2) Replica data completion. When performing replica data completion on a Block, the received data will also be temporarily placed here, and after successful completion, it will be moved into the Finalized directory.

[0056] Block pool directory ( Figure 1A The directory BP-880450491-10.0.0.8-1721802325640 contains the subdirectories current and tmp. The tmp directory is used to store copies of the Temporary state. Under the current directory, the Finalized directory stores copies of the Finalized state, and the rbw directory stores copies of the RBW, RWR, and RUR states.

[0057] As can be seen, all data files reside in the Finalized and rbw directories, and no data files with the same Block ID exist on the same DataNode. Therefore, the goal of reducing capacity is achieved by migrating the blk files from one disk to another, and then unmounting the original disk.

[0058] There are currently two solutions for scaling down HDFS cluster storage: The first is to reduce the number of DataNodes. If the number of DataNodes is large, and it's permissible to reduce CPU and memory resources while reducing storage space, then several DataNodes can be removed, directly utilizing HDFS's decommissioning feature. This is the most common method. However, the scaling process involves migrating a large amount of data across nodes, which can generate significant internal network traffic and potentially impact online load. Close monitoring and manual optimization by operations personnel are necessary. This method cannot be used if the cluster has only three DataNodes remaining, or if the CPU or memory resources cannot be reduced synchronously.

[0059] The second method involves reducing the number of disk directories on individual DataNodes. This means reducing the disk space on each node while keeping the number of DataNodes constant. This can be done by modifying the `dfs.datanode.data.dir` parameter on the DataNode, deleting one or more disk directories, and then waiting for HDFS to automatically replenish the replicas. This method also results in a significant amount of data movement between nodes, generating substantial internal network traffic that may impact online load. Close monitoring and manual optimization by operations personnel are necessary. Furthermore, if there are only two data replicas, it is relatively risky. If a node fails or a disk fails while a disk directory is being deleted, data loss is highly likely.

[0060] Disk replacement typically involves replacing older, less efficient disks with newer, more stable ones, or replacing smaller disks with larger ones. Over long-term operation, due to inherent disk limitations or frequent read / write operations, some hard drives may experience increased latency and slower read / write speeds. Distributed storage architectures commonly suffer from the "slow disk" phenomenon: when a disk experiences performance failure or excessive I / O load leading to increased latency, almost all user services will be severely impacted. The "slow disk phenomenon" can cause a decline in the performance of the entire cluster, and in severe cases, even lead to data loss and service unavailability. The slow disk phenomenon is analogous to the ancient "barrel effect": the maximum capacity of a barrel is determined not by its tallest stave, but by its shortest. Similarly, the performance of distributed storage is not determined by the fastest disk, but by the slowest.

[0061] Disk replacement method: Format and mount the hard drive to be added. Modify the dfs.datanode.data.dir setting in the hdfs-site.xml configuration file on the DataNode node, setting it to the directory to be used from now on (add a new mount point and delete outdated mount points). Similar to the second method of scaling down, configuring the new disk to remove the old disk, waiting for HDFS to automatically replenish replicas, will also result in a large amount of data movement between nodes.

[0062] To address some or all of the technical problems existing in the aforementioned related technologies, embodiments of this application provide a data migration method, apparatus, device, medium, and product based on HDFS block status. The method first sets the source disk into maintenance mode, then statistically analyzes the replica status of block files on the source disk, generating a task list composed of block files corresponding to preset replica statuses. Next, it obtains the storage volume and capacity of the source disk and other disks on the data node, calculates the expected storage capacity of the other disks, and further, if the expected storage capacity of the other disks is greater than the storage capacity of the source disk, migrates the block files in the task list to the other disks according to the priority order corresponding to the preset replica status. Thus, this application sets the maintenance mode at the disk level, avoiding adverse effects on other operations of the data node; by calculating the expected storage capacity of other disks, it assesses whether other disks can carry the data on the source disk, improving the operation and maintenance capabilities of HDFS disks, realizing intra-node data migration, and avoiding internal network traffic generated to replenish replicas when replicas are missing, thereby reducing the risk of cluster service disruptions.

[0063] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0064] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0065] This application provides a data migration method based on HDFS block state, which can be implemented using an HDFS block state-based data migration device or electronic device. The electronic device includes, but is not limited to, in-vehicle terminals, servers, personal computers, laptops, tablets, and smartphones. The operating system of the electronic device can include Android, Apple's iOS, Microsoft's Windows operating system, etc., and this application does not limit this. The electronic device can operate independently to implement this application, or it can connect to a network and implement this application through interactive operation with other computer devices on the network. The network where the electronic device is located includes, but is not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).

[0066] It should be noted that the scope of protection of the data migration method based on HDFS block state described in this application embodiment is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, deleting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0067] The method provided in this application is applicable to disk shrinking and disk replacement scenarios. Disk shrinking can involve unmounting a disk and copying its data to other disks on the same data node. Disk replacement involves first copying the data from a disk to a newly added disk on the same data node, and then unmounting the disk.

[0068] like Figure 2 As shown, Figure 2 This is a flowchart illustrating a data migration method based on HDFS block state according to an embodiment of this application. This method can be executed by a data migration device based on HDFS block state, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 2 As shown, the method mainly includes the following steps S201 to S204:

[0069] S201. Set the source disk to enter maintenance mode.

[0070] A source disk (volume) is a disk that needs to be unloaded, and the data on it needs to be migrated. It can be regarded as a data source and is therefore defined as a source disk.

[0071] For example, Figure 3A As shown, modify the code in org.apache.hadoop.hdfs.server.datanode.datanode to add the configuration item DFS_DATANODE_DATA_DIR_IN_MAINTENANCE_KEY. For example... Figure 3B As shown, add and modify the configuration item `dfs.datanode.data.dir.in.maintenance` in the `hdfs-size.xml` file to set a specific disk in the data node to enter maintenance mode. Figure 3C As shown, after adding or modifying configuration items, the command to dynamically refresh the data node configuration is `hdfsdfsadmin-reconfigdatanodeip:portstart`, where `ip` is the data node to which the source disk belongs. It should be noted that the above modifications only apply to the data nodes; the NameNode is unaware of them and they do not affect the data nodes' ability to provide services. Compared to the community code's ability to set nodes to enter maintenance mode, setting disk-level maintenance mode allows for more granular disk-level operations and maintenance.

[0072] Step S201 sets a disk on a single data node into maintenance mode without affecting write and read operations within the node. By setting the source disk into maintenance mode, new data is skipped from the source disk, allowing it to retain existing block files and preventing new block files from being created, thus mitigating data loss to some extent.

[0073] S202. Calculate the copy status of block files in the source disk and generate a task list.

[0074] The task list includes block files corresponding to the preset copy status.

[0075] It should be noted that the replica state (ReplicaState) of block files in the source disk includes the following three types:

[0076] (1) The current / Finalized directory is in the Finalized state;

[0077] This state includes the following different forms:

[0078] ① Once the preset size is reached, no further append operations will be performed; the preset size is configured via ${dfs.blocksize}, and is typically 128MB by default in Hadoop 2.x and later versions.

[0079] ②If the preset size is not reached, the append operation can be performed.

[0080] ③ Files in a Finalized state but awaiting deletion; for example, Finalized blk files that need to be deleted due to a TRUNCATE operation. This application does not consider this type of file.

[0081] (2) The current / rbw directory is in the RBW, RWR, or RUR state;

[0082] (3) The tmp directory is in Temporary state.

[0083] In this embodiment, the preset replica states include: a first replica state, indicating that it is in the Finalized state and can perform append write operations; a second replica state, indicating that it is in the Finalized state and no longer performs append write operations; and a third replica state, indicating that it is in the replica writing state. Optionally, the third replica state includes: the replica being written under the subdirectory tmp or current / rbw of the block pool directory, or the replica being written under the rebalancing or patching subdirectory tmp of the block pool directory and the replica being written under the subdirectory current / rbw of the block pool directory. current / rbw refers to the rbw directory under the subdirectory current of the block pool directory.

[0084] Accordingly, the task list includes: block files corresponding to the first replica state, block files corresponding to the second replica state, and block files corresponding to the third replica state. Specifically, these include block files in the Finalized state that can be appended to, block files in the Finalized state that are no longer subject to append operations, block files in the tmp subdirectory or current / rbw subdirectory of the block pool directory that are currently being written to, or block files in the tmp subdirectory of the block pool directory that are currently being rebalanced or patched and block files in the current / rbw subdirectory of the block pool directory that are currently being written to.

[0085] For example, the task list includes: blk files that have reached the ${dfs.blocksize} configured size and will not be written again, and their blk_xxx.meta files; blk files smaller than the ${dfs.blocksize} configured size; block files in the tmp or current / rbw subdirectory of the block pool directory that are in the state of being written by replicas, or block files in the tmp subdirectory of the block pool directory that are in the state of being written by replicas during rebalancing or patching, and block files in the current / rbw subdirectory of the block pool directory that are in the state of being written by replicas.

[0086] In step S202, the block pool directories in the source disk are scanned, the replica status is collected and organized, the block files are classified according to the preset replica status, and a task list is generated.

[0087] S203. Obtain the storage capacity and storage amount of the source disk and other disks, and calculate the expected storage capacity of other disks.

[0088] In some embodiments, the data node includes multiple disks, including at least one other disk in addition to the source disk to be unloaded. The other disks can be the original disks of the data node. For example, in the original state, the data node includes three disks: disk A, disk B, and disk C. Assuming that disk C is the source disk to be unloaded, the other disks include disk A and disk B.

[0089] Other disks are disks on the data node other than the source disk, including the original disks and newly added disks pre-mounted to the data node. For example, in its original state, a data node has three disks: disk A, disk B, and disk C. Assume disk C is the source disk to be unmounted and will be replaced by the newly added disk D. Other disks include disks A, B, and D. The mount path for new disks can be added via the configuration item `dfs.datanode.data.dir` in the `hdfs-site.xml` file. The storage size of the source disk represents the amount of data to be migrated, which is the total data size of all block files in the task list.

[0090] In some embodiments, the calculation process for calculating the expected storage capacity of other disks is as follows: Calculate the total storage capacity of the data node based on the storage capacity of the source disk and other disks; calculate the expected total capacity of the data node based on the capacity of other disks; calculate the expected storage rate based on the total storage capacity and expected total capacity of the data node; calculate the expected storage capacity of other disks based on the expected storage rate and the capacity of other disks; and calculate the expected storage capacity of other disks based on the expected storage capacity and the storage capacity of other disks. Here, the expected storage capacity is a storage amount that guarantees disk performance.

[0091] For example, in a disk scaling-down scenario, disk A has a storage capacity of 0.4TB and a capacity of 2TB; disk B has a storage capacity of 0.8TB and a capacity of 2TB; disk C has a storage capacity of 0.6TB and a capacity of 1TB. Disk C is the source disk and needs to be unloaded. First, based on the storage capacity of each disk, the total storage capacity of the data nodes is calculated to be 1.8TB. Based on the capacity of the other disks excluding the source disk, the total capacity of the data nodes is calculated to be 4TB. Based on the total storage capacity and total capacity of the data nodes, the expected storage rate of the disks is calculated to be 0.45. Based on the expected storage rate and the capacity of other disks, the expected storage capacity of other disks is calculated: the expected storage capacity of disk A is 0.9TB, the expected storage capacity of disk B is 0.9TB, and the expected storage capacity of disk C is 0 since it needs to be unloaded. Based on the expected storage capacity and the current storage capacity of other disks, the expected available storage capacity of other disks is calculated: the expected available storage capacity of disk A is 0.5TB, the expected available storage capacity of disk B is 0.1TB, and the expected available storage capacity of disk C is 0 since it needs to be unloaded. As shown in Table 1.

[0092] Table 1

[0093]

[0094] For example, in a disk replacement scenario, disk A has a storage capacity of 0.4TB and a capacity of 2TB; disk B has a storage capacity of 0.8TB and a capacity of 2TB; disk C has a storage capacity of 0.6TB and a capacity of 1TB. Disk C is the source disk and needs to be unloaded. A new disk D with a capacity of 2TB is added to the data node. First, based on the storage capacity of each disk, the total storage capacity of the data node is calculated to be 1.8TB. Based on the capacity of the disks other than the source disk, the total capacity of the data node is calculated to be 6TB. Based on the total storage capacity and the total storage capacity of the data node, the expected storage rate of the disks is calculated to be 0.3. Based on the expected storage rate and the capacity of other disks, the expected storage capacity of the other disks is calculated: the expected storage capacity of disk A is 0.6TB, the expected storage capacity of disk B is 0.6TB, the expected storage capacity of disk D is 0.6TB, and the expected storage capacity of disk C is 0 if it needs to be unloaded. Based on the expected storage capacity and the current storage capacity of the other disks, the expected available storage capacity of the other disks is calculated: the expected available storage capacity of disk A is 0.2TB, the expected available storage capacity of disk B is -0.2TB, the expected available storage capacity of disk D is 0.6TB, and the expected available storage capacity of disk C is 0 if it needs to be unloaded. See Table 2.

[0095] Table 2

[0096]

[0097] The above embodiments obtain the storage amount and capacity of each disk on the data node, clarify the used space and actual space of each disk, and if the source disk needs to be unloaded, the space of the source disk will no longer be used. The total storage of the data node remains unchanged but the total capacity decreases. By calculating the expected storage capacity of other disks, it is determined whether other disks on the data node can carry the data on the source disk, so as to improve the operation and maintenance capabilities of HDFS disks.

[0098] In some embodiments, the reserved storage capacity of other disks can also be obtained, and the maximum storage capacity can be calculated by combining the capacity of other disks. Then, it can be determined whether the storage capacity of the source disk is less than or equal to the maximum storage capacity of other disks.

[0099] The reserved storage capacity can be set through configuration options (dfs.disk.replacer.max.diskreserved or dfs.disk.replacer.max.diskreserved.pct). The maximum storage capacity is the upper limit of the disk; the disk storage capacity can exceed the expected storage capacity, but cannot exceed the maximum storage capacity.

[0100] For example, disk A has a reserved storage capacity of 0.2TB and a total capacity of 2TB, so the maximum storage capacity of disk A is calculated to be 1.8TB.

[0101] In the above embodiment, by further obtaining the reserved storage amount of other disks on the data node besides the source disk, the maximum storage amount of other disks is calculated, and the maximum amount of data that other disks can carry is determined. Based on the maximum storage amount, it is then determined whether other disks on the data node can completely carry the data on the source disk.

[0102] S204. If the expected storage capacity of other disks is greater than the storage capacity of the source disk, migrate the block files in the task list to other disks in order of priority.

[0103] The priority order corresponds to the preset replica states. Optionally, the first replica state has the highest priority, the second replica state has the next highest priority, and the third replica state has the lowest priority. That is, the highest priority is when the replica is in the Finalized state and can perform append write operations; the second highest priority is when the replica is in the Finalized state and no longer performs append write operations; and the lowest priority is when the rbw or tmp directory is in the RBW state.

[0104] If the expected storage capacity of other disks is greater than that of the source disk, it means that, apart from the source disk that needs to be unloaded, the other disks on the current data node have sufficient available space to store the data from the source disk, and these other disks can be used effectively. Then, the block files in their pre-defined replica states, which have already been organized on the source disk, are migrated to the other disks in order of priority, completing the data migration.

[0105] For example, if the expected storage capacity of other disks is greater than the storage capacity of the source disk, first migrate the block files in the Finalized state that can be appended to other disks, then migrate the block files in the Finalized state that are no longer appended to other disks, and finally migrate the block files in the copy writing state under the subdirectory tmp or current / rbw of the block pool directory, or the block files in the copy writing state under the rebalancing or patching state under the subdirectory tmp of the block pool directory and the block files in the copy writing state under the subdirectory current / rbw of the block pool directory to other disks.

[0106] In some embodiments, when performing step S204, if the expected storage capacity of other disks is greater than the storage capacity of the source disk, the block file corresponding to the first replica state is first migrated to other disks, and then the block file corresponding to the second replica state is migrated to other disks; then, for the block file corresponding to the third replica state, after a preset waiting time, the target block file is migrated to other disks; wherein, the target block file is the block file in the block file corresponding to the third replica state that has been converted to the Finalized state.

[0107] Specifically, if the expected storage capacity of other disks is greater than that of the source disk, block files in the Finalized state that can be appended to other disks are migrated first, according to priority. Then, block files in the Finalized state that are no longer subject to append operations are migrated to other disks. Next, block files in the tmp or current / rbw directories that were previously in a write state but have now been converted to the Finalized state are migrated to other disks. For block files in the tmp or current / rbw directories that are in a write state, after waiting for a preset waiting time (default 15 minutes), block files that have completed writing and been converted to the Finalized state are migrated to other disks. Block files that have not completed writing after the preset waiting time will remain on the source disk.

[0108] In some embodiments, when performing step S204, there are multiple other disks, and the block file will be split and migrated to multiple other disks. To avoid disk storage imbalance, a tool (such as DiskBalancer) can be used to balance the disks. As shown in Table 2, 0.2TB of data on disk B can be migrated to disk A.

[0109] The above embodiments are applied in disk replacement scenarios. Data nodes can function normally, and read operations on the source disk are not affected. Newly added disks can be put into use immediately, sharing the storage and read / write pressure. This avoids the impact of a large amount of patching traffic caused by missing replicas on cluster services.

[0110] In some embodiments, after executing step S203, if the expected storage capacity of other disks is less than the storage capacity of the source disk, the reserved storage capacity of other disks is obtained, and the maximum storage capacity of other disks is calculated by combining the storage capacity of other disks; it is then determined whether the maximum storage capacity of other disks is greater than the storage capacity of the source disk. If so, step S204 is executed to migrate the block files in the task list to other disks in priority order.

[0111] For example, suppose the data nodes only include disks A and C, and disk C needs to be unloaded. After the above steps, it is determined that the expected storage capacity of disk A (0.9TB) is less than the storage capacity of the source disk (1TB), indicating that disk A may not achieve an ideal state if used as the target disk. Therefore, the reserved storage capacity of disk A (0.2TB) is obtained. Combined with the storage capacity of disk A (2TB), the maximum storage capacity of disk A is calculated to be 1.8TB. Then, it is determined whether the maximum storage capacity of disk A is greater than the storage capacity of the source disk to estimate whether disk A can accommodate all the data from the source disk. If it is determined that 1.8TB is greater than the storage capacity of the source disk (1TB), it means that although disk A cannot be in an ideal state after accommodating all the data from the source disk, it can still meet the requirement of unloading disk C for scaling down. In this case, all block files in the task list can be migrated to disk A in priority order.

[0112] If it is determined that the maximum storage capacity of other disks is greater than that of the source disk, it means that the other disks are sufficient to store the data on the source disk. The block files in the task list corresponding to the source disk that were previously organized can be migrated to other disks in order of priority.

[0113] The above embodiments calculate the expected storage capacity and maximum storage capacity of other disks to ensure that the data storage capacity on other disks after data migration does not exceed the maximum storage capacity, or even the expected storage capacity. Thus, by unloading or replacing the source disk, the normal operation of other disks is not affected, and there will be no adverse impact on the cluster business.

[0114] When it is determined that the maximum storage capacity of other disks is less than the storage capacity of the source disk, it indicates that the space on other disks on this data node is insufficient to store the data on the source disk. In this embodiment, an optional implementation method is provided, including: when the maximum storage capacity of other disks is less than the storage capacity of the source disk, determining whether to initiate cross-node migration; if so, migrating the block files in the task list to other disks in priority order until the storage capacity of other disks reaches its maximum storage capacity, and then migrating the remaining block files to the disks of other data nodes.

[0115] Specifically, when the disk within a single node is insufficient to store the unloaded disk, the cross-node parameter (crossdn) is queried first to determine whether cross-node migration is enabled. If the cross-node parameter indicates that it is enabled, the data migration between disks is completed within the node first, and then the remaining block files are migrated to the disks of other data nodes.

[0116] After completing the data migration using the above embodiments, set the source disk to exit maintenance mode and modify the configuration item dfs.datanode.data.dir in the configuration file hdfs-site.xml to remove the source disk from the data node.

[0117] In the above embodiments, data migration occurs within the data node and does not generate cross-node traffic, thus avoiding impact on cluster services.

[0118] In some embodiments, the data migration method provided in this application can be executed via command line. After setting the source disk to enter maintenance mode, the following command is executed:

[0119] Execute the Plan command: `hdfs diskreplacer-plan datanodename` to generate the execution plan. For example... Figure 4A As shown, Figure 4A This shows the parameters, default values, and descriptions included in the Plan command.

[0120] Then execute the Execute command: hdfs diskreplacer-execute<JSON file path> Migrate block files according to priority. For example... Figure 4B As shown, Figure 4B This shows the parameters, default values, and descriptions included in the Execute command.

[0121] Then execute the query command hdfs diskreplacer-query <datanode>The current execution progress of the HDFS disk replacer is obtained from the data nodes of the running plan to query the data migration progress, such as... Figure 4C As shown, Figure 4C This displays the parameters, default values, and descriptions included in the Query command.

[0122] After the data migration is complete, set the source disk to exit maintenance mode and modify the configuration to unload the source disk.

[0123] During the process, the Cancel command hdfs diskreplacer-cancel can be executed.<JSON file path> Cancel the operation plan. For example... Figure 4D As shown, Figure 4D This shows the explanation of the Cancel command.

[0124] The above command line execution requires the addition of diskreplacer-related configuration items in the hdfs-site.xml file, as shown in Table 3.

[0125] Table 3

[0126]

[0127] The above embodiments complete data migration through operation instructions provided by the command line, and can also intuitively provide feedback on the execution progress, which is convenient for operation and maintenance personnel and realizes fine-grained single-node disk-level storage management.

[0128] In summary, this application provides a data migration method based on HDFS block status. The method first sets the source disk into maintenance mode, then calculates the replica status of block files on the source disk, generating a task list consisting of block files corresponding to preset replica statuses. Next, it obtains the storage volume and capacity of the source disk and other disks on the data node, calculates the expected storage capacity of the other disks, and further, if the expected storage capacity of the other disks is greater than the storage capacity of the source disk, migrates the block files in the task list to the other disks according to their priority order corresponding to the preset replica status. Thus, this application sets a maintenance mode at the disk level, avoiding adverse effects on other operations of the data node; by calculating the expected storage capacity of other disks, it assesses whether other disks can carry the data on the source disk, improving the operation and maintenance capabilities of HDFS disks, realizing intra-node data migration, and avoiding internal network traffic generated to replenish replicas when replicas are missing, thereby reducing the risk of cluster service disruptions.

[0129] like Figure 5 As shown, Figure 5 This application provides a data migration apparatus based on HDFS block state, the apparatus comprising:

[0130] Setting module 501 is used to set the source disk to enter maintenance mode;

[0131] The statistics module 502 is used to count the replica status of block files in the source disk and generate a task list; the task list includes block files corresponding to preset replica status.

[0132] The calculation module 503 is used to obtain the storage quantity and capacity of the source disk and other disks, and to calculate the expected storage capacity of the other disks; the source disk and other disks belong to the same data node;

[0133] Migration module 504 is used to migrate block files in the task list to other disks in priority order when the expected storage capacity of other disks is greater than the storage capacity of the source disk. The priority order corresponds to the preset copy status.

[0134] As an optional implementation of this application, the calculation module 503 is specifically used for: calculating the total storage of the data node based on the storage capacity of the source disk and other disks; calculating the expected total capacity of the data node based on the capacity of other disks; calculating the expected storage rate based on the total storage of the data node and the expected total capacity; calculating the expected storage capacity of other disks based on the expected storage rate and the capacity of other disks; and calculating the expected storable capacity of other disks based on the expected storage capacity of other disks and the storage capacity of other disks.

[0135] As an optional implementation of this application, the calculation module 503 is further configured to: obtain the reserved storage capacity of other disks when the expected storage capacity of other disks is less than the storage capacity of the source disk; calculate the maximum storage capacity of other disks based on the reserved storage capacity of other disks and the storage capacity of other disks; the migration module 504 is further configured to migrate the block files in the task list to other disks in priority order when the maximum storage capacity of other disks is greater than the storage capacity of the source disk.

[0136] As an optional implementation of this application, the calculation module 503 is further configured to: determine whether to migrate across nodes when the maximum storage capacity of other disks is less than the storage capacity of the source disk; the migration module 504 is further configured to: if so, migrate the block files in the task list to other disks in priority order until the storage capacity of other disks reaches the maximum storage capacity, and then migrate the remaining block files across nodes to the disks of other data nodes.

[0137] As an optional implementation of this application, the preset replica states include: a first replica state, indicating that the replica has been written to the Finalized state and can be appended; a second replica state, indicating that the replica is in the Finalized state and no further appending operations are being performed; and a third replica state, indicating that the replica is being written to.

[0138] As an optional implementation of this application, the migration module 504 is specifically used to: migrate the block file corresponding to the first replica state to the other disk when the expected storage capacity of the other disk is greater than the storage capacity of the source disk, and then migrate the block file corresponding to the second replica state to the other disk; for the block file corresponding to the third replica state, after a preset waiting time, migrate the target block file to the other disk; wherein, the target block file is the block file in the block file corresponding to the third replica state that has been converted to the Finalized state.

[0139] Specific limitations regarding the HDFS block state-based data migration device can be found in the limitations of the HDFS block state-based data migration method described above, and will not be repeated here. Each module in the aforementioned HDFS block state-based data migration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0140] In one embodiment, this application provides an electronic device, which may be a terminal, and its internal structure diagram may be as follows. Figure 6 As shown, the electronic device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a stuttering detection method. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0141] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0142] In one embodiment, the HDFS block state-based data migration apparatus provided in this application can be implemented as a computer program, which can be implemented in the form of, for example... Figure 6 The device operates on the illustrated electronic device. The electronic device's memory can store various program modules that constitute the HDFS block state-based data migration apparatus. The computer program comprised of these program modules causes the processor to execute the steps of the HDFS block state-based data migration methods described in the various embodiments of this application.

[0143] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by the computer program, implements the steps of the HDFS block state-based data migration method described in the various embodiments of this application.

[0144] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0145] It should be understood, in the several embodiments provided in this application, that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0146] In this application, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0147] In this application, memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0148] In this application, computer-readable media includes both permanent and non-permanent, removable and non-removable storage media. Storage media can implement information storage using any method or technology; the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0149] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0150] The above are merely specific embodiments of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.< / datanode>

Claims

1. A data migration method based on HDFS block state, characterized in that, include: Set the source disk to enter maintenance mode; The replica status of block files in the source disk is statistically analyzed to generate a task list; wherein, the replica status is used to display the current status and visibility of the replica data; the task list includes block files corresponding to preset replica statuses; the preset replica statuses include: a first replica status, indicating that the replica has been written to the Finalized state and can be appended; a second replica status, indicating that the replica is in the Finalized state and no further appending operations are being performed; and a third replica status, indicating that the replica is being written to. Obtain the storage volume and capacity of the source disk and other disks, and calculate the expected storage capacity of the other disks; the source disk and the other disks belong to the same data node; If the expected storage capacity of the other disks is greater than the storage capacity of the source disk, the block files in the task list are migrated to the other disks in priority order, which corresponds to the preset copy status. The step of obtaining the storage capacity and amount of the source disk and other disks, and calculating the expected storage capacity of the other disks, includes: Calculate the total storage capacity of the data node based on the storage capacity of the source disk and other disks; Calculate the expected total capacity of the data node based on the capacity of the other disks; Calculate the expected storage rate based on the total storage capacity and expected total capacity of the data nodes; Calculate the expected storage capacity of the other disks based on the expected storage rate and the capacity of the other disks; Calculate the expected storage capacity of the other disks based on their expected storage capacity and storage capacity.

2. The method according to claim 1, characterized in that, After obtaining the storage capacity and storage amount of the source disk and other disks, and calculating the expected storage capacity of the other disks, the method further includes: If the expected storage capacity of the other disks is less than the storage capacity of the source disk, obtain the reserved storage capacity of the other disks; Calculate the maximum storage capacity of the other disks based on their reserved storage capacity and total storage capacity. If the maximum storage capacity of other disks is greater than the storage capacity of the source disk, the block files in the task list are migrated to the other disks in order of priority.

3. The method according to claim 2, characterized in that, After calculating the maximum storage capacity of the other disks based on their reserved storage capacity and total storage capacity, the method further includes: If the maximum storage capacity of other disks is less than the storage capacity of the source disk, determine whether to enable cross-node migration; If so, the block files in the task list are migrated to the other disks in priority order until the storage capacity of the other disks reaches the maximum storage capacity, and then the remaining block files are migrated across nodes to the disks of other data nodes.

4. The method according to claim 1, characterized in that, When the expected storage capacity of the other disks is greater than the storage capacity of the source disk, migrating the block files in the task list to the other disks in priority order includes: If the expected storage capacity of the other disks is greater than the storage capacity of the source disk, the block files corresponding to the first replica state are migrated to the other disks, and then the block files corresponding to the second replica state are migrated to the other disks. For the block file corresponding to the third replica state, after a preset waiting time, the target block file is migrated to the other disk; wherein, the target block file is the block file in the block file corresponding to the third replica state that has been converted to the Finalized state.

5. A data migration device based on HDFS block state, characterized in that, include: The settings module is used to configure the source disk to enter maintenance mode. The statistics module is used to count the replica status of block files in the source disk and generate a task list. The replica status displays the current status and visibility of the replica data. The task list includes block files corresponding to preset replica statuses. The preset replica statuses include: a first replica status, indicating that the replica has been written to and is in a Finalized state, allowing append operations; a second replica status, indicating that the replica is in a Finalized state and no further append operations are being performed; and a third replica status, indicating that the replica is currently being written to. A calculation module is used to obtain the storage quantity and capacity of a source disk and other disks, and calculate the expected storable quantity of the other disks; the source disk and the other disks belong to the same data node; wherein, obtaining the storage quantity and capacity of the source disk and other disks, and calculating the expected storable quantity of the other disks includes: calculating the total storage of the data node based on the storage quantity of the source disk and other disks; calculating the expected total capacity of the data node based on the capacity of the other disks; calculating the expected storage rate based on the total storage and expected total capacity of the data node; calculating the expected storage quantity of the other disks based on the expected storage rate and the capacity of the other disks; and calculating the expected storable quantity of the other disks based on the expected storage quantity and the storage quantity of the other disks. The migration module is used to migrate block files in the task list to the other disks in priority order, whereby the expected storage capacity of the other disks is greater than the storage capacity of the source disk, and the priority order corresponds to the preset copy status.

6. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the data migration method based on HDFS block state as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, include: A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the data migration method based on HDFS block state as described in any one of claims 1 to 4.

8. A computer program product, characterized in that, include: The computer program product includes a computer program that, when run on a computer, causes the computer to implement the data migration method based on HDFS block state as described in any one of claims 1 to 4.

Images