A data migration method, device, equipment and medium

Abstract

The application provides a data migration method, device, equipment and medium. The method comprises the following steps: determining the data density difference value of each sub-cluster in a federal cluster; determining a first sub-cluster and a second sub-cluster in the federal cluster according to the data density difference value of each sub-cluster in the federal cluster, wherein the data density difference value of the first sub-cluster is greater than a first threshold value, the data density difference value of the second sub-cluster is less than a second threshold value, and the first threshold value is greater than the second threshold value; and migrating the data of the first sub-cluster to the second sub-cluster. According to the method, the data density difference value of each sub-cluster is compared with the first threshold value and the second threshold value, the state of the data density of each sub-cluster is confirmed in real time, the data migration program is triggered according to the state of the data density of each sub-cluster, manual judgment is avoided, the storage data amount between the sub-clusters can be adjusted in time, and the overall use effect of the federal cluster is improved.

Description

Technical Field

[0001] This application relates to the field of data migration technology, and in particular to a data migration method, apparatus, device and medium. Background Technology

[0002] A federated cluster is a distributed cluster management system that includes multiple sub-clusters. Due to differences in business operations, there are access hotspots in various directories or files, leading to storage imbalances among nodes and sub-clusters.

[0003] In existing technologies, data migration between sub-clusters is carried out manually. However, manual balancing often only begins after the business is affected by extreme data imbalance, which impacts the effectiveness of the federated cluster. Summary of the Invention

[0004] This application provides a data migration method, apparatus, device, and medium to solve the problem in the prior art where manual balancing is only started after the business is already affected by extremely unbalanced data, thus affecting the effectiveness of the federated cluster.

[0005] Firstly, a data migration method is provided for application in a data migration system, the method comprising:

[0006] The data density difference in each sub-cluster of the federated cluster is determined based on the target data density and the data density of the sub-cluster. The target data density is determined based on the total storage of the federated cluster and the total storage usage. The data density of the sub-cluster is determined based on the total storage of the sub-cluster and the storage usage of the sub-cluster.

[0007] Based on the data density difference of each sub-cluster in the federated cluster, the first sub-cluster and the second sub-cluster in the federated cluster are determined, wherein the data density difference of the first sub-cluster is greater than a first threshold, the data density difference of the second sub-cluster is less than a second threshold, and the first threshold is greater than the second threshold.

[0008] Migrate the data from the first sub-cluster to the second sub-cluster.

[0009] In this application, the data density difference in each sub-cluster within the federated cluster is determined, including:

[0010] The target data density is obtained by calculating the ratio between the total storage usage and the total storage volume.

[0011] The data density of a sub-cluster is obtained by calculating the ratio between the storage usage of the sub-cluster and the total storage of the sub-cluster.

[0012] The difference between the sub-cluster data density and the target data density is calculated to determine the data density difference in each sub-cluster within the federated cluster.

[0013] In this application, migrating data from the first sub-cluster to the second sub-cluster includes:

[0014] Determine the source mount point of the first sub-cluster, the data to be migrated from the source mount point, and the target mount point of the second sub-cluster;

[0015] If the amount of data to be migrated is greater than the preset amount of data to be migrated, then the data to be migrated will be migrated to the target mount point of the second sub-cluster according to the preset batch migration amount.

[0016] In this application, if the amount of data to be migrated is greater than the preset amount of data to be migrated, the data to be migrated is migrated to the target mount point of the second sub-cluster according to the preset batch migration amount, including:

[0017] If the amount of data to be migrated is greater than the preset amount of data to be migrated, then the data to be migrated will be copied to the target mount point of the second sub-cluster according to the preset batch migration amount, wherein the source mount point is in an unwritable state.

[0018] When the migration of the data to be migrated is completed, determine the migrated data of the target mount point in the second sub-cluster;

[0019] Compare the data to be migrated with the data that has already been migrated to obtain the comparison results;

[0020] Based on the comparison results, the source mount point status is changed from non-writable to writable.

[0021] Delete the data to be migrated from the first sub-cluster.

[0022] In this application, if the amount of data to be migrated is greater than the preset amount of data to be migrated, the data to be migrated is copied to the target mount point of the second sub-cluster according to the preset batch migration amount, including:

[0023] Determine the current batch of data to be migrated;

[0024] Based on the source mount point and the target mount point, the current batch of data to be migrated is copied to the target mount point of the second sub-cluster to obtain the first migrated data;

[0025] After the current batch of data is copied, change the source mount point status to a non-writable state;

[0026] Determine the remaining data to be migrated;

[0027] Based on the source mount point and the target mount point, the remaining data of the data to be migrated is copied to the target mount point of the second sub-cluster to obtain the second migrated data;

[0028] Based on the first migrated data and the second migrated data, the migrated data is obtained.

[0029] In this application, the data to be migrated in the first sub-cluster is deleted, including:

[0030] Determine the mount table corresponding to the data to be migrated;

[0031] The mount table corresponding to the data to be migrated will be updated with the source mount point as the target mount point;

[0032] Set the wait time for deleting data to be migrated;

[0033] Based on the waiting time for deleting the data to be migrated, the data to be migrated is deleted from the first sub-cluster.

[0034] In this application, after determining the source mount point of the first sub-cluster, the data to be migrated from the source mount point, and the target mount point of the second sub-cluster, the method further includes:

[0035] If the amount of data to be migrated is less than or equal to the preset amount of data to be migrated, the source mount point status will be adjusted to an unwritable state.

[0036] Based on the source mount point and the target mount point, the data to be migrated is copied to the target mount point of the second sub-cluster to obtain the migrated data.

[0037] In this application, the method also includes:

[0038] When the data density difference of the first sub-cluster is less than the first threshold, stop migrating data from the first sub-cluster to the second sub-cluster.

[0039] When the difference in cluster data density between the second sub-cluster and the third threshold is greater than the third threshold, the data in the first sub-cluster is migrated to the third sub-cluster. The difference in sub-cluster data density between the third sub-cluster and the third threshold is greater than the fourth threshold.

[0040] In this application, the method further includes: sending real-time execution status information of the data migration system to the state memory according to a preset reporting mechanism, wherein the real-time execution status information is used to characterize the execution status of the data migration method;

[0041] The receiving state memory generates operation instructions based on real-time execution status information. The operation instructions include a continue execution instruction indicating that the execution status of the data migration method is normal, and a stop execution instruction indicating that the execution status of the data migration method is abnormal.

[0042] Control the data migration system according to the operating instructions.

[0043] Secondly, this application provides a data migration apparatus for use in a data migration system, the apparatus comprising:

[0044] The first determining module is used to determine the data density difference in each sub-cluster of the federated cluster. The data density difference of the sub-cluster is determined based on the target data density and the sub-cluster data density of the sub-cluster. The target data density is determined based on the total storage of the federated cluster and the total storage usage. The sub-cluster data density is determined based on the total storage of the sub-cluster and the sub-cluster storage usage of the sub-cluster.

[0045] The second determining module is used to determine the first sub-cluster and the second sub-cluster in the federated cluster based on the data density difference of each sub-cluster in the federated cluster, wherein the data density difference of the first sub-cluster is greater than a first threshold, the data density difference of the second sub-cluster is less than a second threshold, and the first threshold is greater than the second threshold.

[0046] The migration module is used to migrate data from the first sub-cluster to the second sub-cluster.

[0047] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0048] The memory stores the instructions that the computer executes;

[0049] The processor executes computer-executable instructions stored in memory to implement any of the aforementioned data migration methods.

[0050] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the aforementioned data migration methods.

[0051] This application provides a data migration method, apparatus, device, and medium. It determines the data density difference between each sub-cluster in a federated cluster. The data density difference is determined based on a target data density and the data density of the sub-clusters within the federated cluster. The target data density is determined based on the total storage capacity and overall storage usage of the federated cluster. The sub-cluster data density is determined based on the total storage capacity and storage usage of the sub-clusters within the sub-clusters. Based on the data density difference between each sub-cluster in the federated cluster, a first sub-cluster and a second sub-cluster are identified. The data density difference of the first sub-cluster is greater than a first threshold, and the data density difference of the second sub-cluster is less than a second threshold. The first threshold is greater than the second threshold. The method then migrates data from the first sub-cluster to the second sub-cluster. This application compares the data density difference of each sub-cluster with the first and second thresholds to confirm the data density status of each sub-cluster, thereby determining which sub-clusters need data migration out and which need data migration in. When the data volume of a sub-cluster exceeds the first threshold, a data migration procedure is triggered, avoiding manual judgment and enabling timely adjustment of the storage data volume between sub-clusters, thus improving the overall efficiency of the federated cluster. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0053] Figure 1 A schematic diagram illustrating a data migration scenario provided in this application;

[0054] Figure 2 A flowchart illustrating a data migration method provided in this application;

[0055] Figure 3 A flowchart illustrating another data migration method provided in this application;

[0056] Figure 4 A schematic diagram of the structure of a data migration device provided in this application;

[0057] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0058] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0059] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0060] To clearly understand the technical solution of this application, the solutions of the prior art will be described in detail first.

[0061] A federated cluster is a distributed cluster management system that includes multiple sub-clusters. Due to differences in business operations, there are access hotspots in various directories or files, leading to storage imbalances among nodes and sub-clusters.

[0062] In existing technologies, data migration between sub-clusters is carried out manually. However, manual balancing often only begins after the business is affected by extreme data imbalance, which impacts the effectiveness of the federated cluster.

[0063] To address the issue of manually balancing data only after it has severely impacted business operations, thus affecting the effectiveness of the federated cluster, the inventors discovered that by using the federated cluster data density as a reference value, the difference between the data density of the sub-clusters and the data density of the federated cluster can be calculated. By setting a first threshold and a second threshold to characterize the safe range of the data density difference between the sub-clusters, sub-clusters with data density exceeding this safe range are identified as sub-clusters requiring data migration, while sub-clusters with data density below this safe range are identified as sub-clusters that can receive data. This achieves the goal of automatically triggering sub-cluster data migration, avoiding impact on the business operations of the sub-clusters, and thus improving the effectiveness of the federated cluster.

[0064] The following describes the application scenarios of the data migration method provided in the embodiments of this application.

[0065] Figure 1 This application provides a schematic diagram of a data migration scenario, such as... Figure 1 As shown, this scenario includes a federated cluster and a data migration system. The federated cluster comprises multiple sub-clusters. To maintain data balance among the sub-clusters, the data migration system calculates the data density difference of each sub-cluster at preset intervals to ensure it falls within a first and second threshold range. When the data density difference of a sub-cluster exceeds the first threshold, the data migration program is initiated, designating that sub-cluster as the first sub-cluster. Then, a second sub-cluster with a data density difference below the second threshold is identified, and data from the first sub-cluster is migrated to the second sub-cluster. This automatically balances the data within the federated cluster, reducing the impact on business operations and improving the effectiveness of the sub-clusters.

[0066] Among them, the federated cluster can refer to the HDFS (Hadoop Distributed File System) RBF (Router Bbased Federation) cluster. The HDFS RBF cluster achieves the effect of combining multiple sub-clusters, solving the problem of limited cluster storage capacity. It can distribute data storage and complete the efficient storage and management of massive amounts of data.

[0067] A sub-cluster can refer to a member cluster in a federated cluster, such as an HDFS cluster, a regular federated cluster, other federated clusters, or a hybrid cluster of independent clusters.

[0068] It should be noted that the above application scenarios are merely illustrative. For example, the data migration system can also be any other system with data migration functionality. The data migration methods, apparatus, devices, and media provided in the embodiments of this application include, but are not limited to, the above application scenarios.

[0069] Figure 2 A flowchart illustrating a data migration method provided in this application is shown below. Figure 2 As shown, the method includes:

[0070] S201. Determine the data density difference in each sub-cluster of the federated cluster. The data density difference of the sub-cluster is determined based on the target data density and the sub-cluster data density of the sub-cluster. The target data density is determined based on the total storage of the federated cluster and the total storage usage. The sub-cluster data density is determined based on the total storage of the sub-cluster and the sub-cluster storage usage.

[0071] Data density can refer to storage utilization. In this embodiment, data density is the ratio of total storage usage to total storage capacity. Total storage usage refers to all used storage in a sub-cluster or federated cluster, and total storage capacity refers to the total available storage capacity of the sub-cluster or federated cluster. For example, if a sub-cluster has an available storage capacity of 10T and used storage of 1T, then the total storage usage of the sub-cluster is 1T, and the total storage capacity is 10T. The data density of this sub-cluster is 1T / 10T = 0.1.

[0072] The data density difference refers to the difference between the data density of a sub-cluster and the target data density. In this embodiment, the target data density is the data density of the federated cluster. The data density difference can be positive or negative. A positive value indicates that the data density of the sub-cluster is higher than the target data density, while a negative value indicates that the data density of the sub-cluster is lower than the target data density. By calculating the data density difference of the sub-cluster, the degree to which the data density of the sub-cluster is higher or lower than the target data density can be determined, thus providing data support for subsequent steps to determine whether the sub-cluster needs data migration.

[0073] Methods for determining the data density difference in each sub-cluster of a federated cluster may include: periodically collecting the total storage and overall storage usage of each sub-cluster in the federated cluster through a data migration system, and calculating the data density difference of each sub-cluster.

[0074] In this embodiment of the application, determining the data density difference in each sub-cluster within the federated cluster includes:

[0075] The target data density is obtained by calculating the ratio between the total storage usage and the total storage volume.

[0076] The data density of a sub-cluster is obtained by calculating the ratio between the storage usage of the sub-cluster and the total storage of the sub-cluster.

[0077] The difference between the sub-cluster data density and the target data density is calculated to determine the data density difference in each sub-cluster within the federated cluster.

[0078] S202. Based on the data density difference of each sub-cluster in the federated cluster, determine the first sub-cluster and the second sub-cluster in the federated cluster, wherein the data density difference of the first sub-cluster is greater than a first threshold, the data density difference of the second sub-cluster is less than a second threshold, and the first threshold is greater than the second threshold.

[0079] The first sub-cluster can refer to the sub-cluster whose data density difference is greater than the first threshold. When the data density difference of the sub-cluster is greater than the first threshold, it means that the data density of the sub-cluster exceeds the preset range and data migration is required.

[0080] The second sub-cluster can refer to a sub-cluster whose data density difference is less than the second threshold. When the data density difference of a sub-cluster is less than the second threshold, it means that the data density of the sub-cluster is lower than the preset range, and there is still a lot of storage capacity available for data storage. Therefore, it can receive the data migrated out of the first sub-cluster.

[0081] The first threshold can refer to the upper limit of the preset range, and the second threshold can refer to the lower limit of the preset range. For example, if the preset range is -10 to 10, then the first threshold is 10 and the second threshold is -10.

[0082] The method for determining the first and second sub-clusters in a federated cluster can include: when the data migration system detects that the data density difference of a sub-cluster in the federated cluster exceeds a first threshold, that sub-cluster is designated as the first sub-cluster. If the data density difference of other sub-clusters in the federated cluster is lower than a second threshold, those other sub-clusters can be designated as the second sub-clusters. Alternatively, when there are multiple other sub-clusters, any one of them can be selected as the second sub-cluster, or the sub-cluster with the smallest data density difference among the other sub-clusters can be selected as the second sub-cluster. If no other sub-cluster in the federated cluster has a data density difference lower than the second threshold, the administrator can be notified to perform data migration by adding a sub-cluster or by manually selecting a second sub-cluster.

[0083] S203. Migrate the data from the first sub-cluster to the second sub-cluster.

[0084] Data migration can refer to moving data from the first sub-cluster to the second sub-cluster. After this is completed, the migrated data in the first sub-cluster will no longer exist.

[0085] In this embodiment of the application, migrating data from the first sub-cluster to the second sub-cluster includes:

[0086] Determine the source mount point of the first sub-cluster, the data to be migrated from the source mount point, and the target mount point of the second sub-cluster;

[0087] If the amount of data to be migrated is greater than the preset amount of data to be migrated, then the data to be migrated will be migrated to the target mount point of the second sub-cluster according to the preset batch migration amount.

[0088] The mount point can refer to the mount point in the Linux (operating system kernel) file system. The mount point is the entry directory of the disk file system in Linux. In this embodiment, the source mount point is the mount point path of the first sub-cluster, and the target mount point is the mount point path of the second sub-cluster.

[0089] The data to be migrated can refer to the data in the mount point path. In this embodiment, it specifically refers to the data in the source mount point path. For example, if the source mount point is: first sub-cluster / data, then the data to be migrated is the data in the first sub-cluster / data.

[0090] The method for determining the source mount point may include: determining the amount of data exceeding a preset range based on the data density difference of the first sub-cluster, then selecting a mount point from the first sub-cluster whose data amount is greater than the amount of data exceeding the preset range, and after determining that there are no sub-mount points under the mount point, using the mount point as the source mount point.

[0091] Methods for determining the data to be migrated include: based on the determined source mount point, identifying the data in the source mount point as the data to be migrated.

[0092] The method for determining the target mount point includes: creating an empty directory in the second sub-cluster with the same path as the source mount point. For example, if the source mount point is: first sub-cluster / data, then the target mount point is: second sub-cluster / data.

[0093] In this embodiment of the application, if the amount of data to be migrated is greater than a preset amount of data to be migrated, the data to be migrated is migrated to the target mount point of the second sub-cluster according to the preset batch migration amount, including:

[0094] If the amount of data to be migrated is greater than the preset amount of data to be migrated, then the data to be migrated will be copied to the target mount point of the second sub-cluster according to the preset batch migration amount, wherein the source mount point is in an unwritable state.

[0095] When the migration of the data to be migrated is completed, determine the migrated data of the target mount point in the second sub-cluster;

[0096] Compare the data to be migrated with the data that has already been migrated to obtain the comparison results;

[0097] Based on the comparison results, the source mount point status is changed from non-writable to writable.

[0098] Delete the data to be migrated from the first sub-cluster.

[0099] The "unwritable" state can refer to a state where data cannot be written to the source mount point.

[0100] One method to compare the data to be migrated with the data that has already been migrated is to use the diff function in Hadoop's built-in distributed replication program distcp to generate a snapshot difference report to check whether the data at the source mount point and the target mount point are consistent.

[0101] Methods for changing the state of a source mount point from non-writable to writable may include: adjusting the configuration data of the source mount point to change its state from non-writable to writable.

[0102] In this embodiment of the application, if the amount of data to be migrated is greater than a preset amount of data to be migrated, then the data to be migrated is copied to the target mount point of the second sub-cluster according to the preset batch migration amount, including:

[0103] Determine the current batch of data to be migrated;

[0104] Based on the source mount point and the target mount point, the current batch of data to be migrated is copied to the target mount point of the second sub-cluster to obtain the first migrated data;

[0105] After the current batch of data is copied, change the source mount point status to a non-writable state;

[0106] Determine the remaining data to be migrated;

[0107] Based on the source mount point and the target mount point, the remaining data of the data to be migrated is copied to the target mount point of the second sub-cluster to obtain the second migrated data;

[0108] Based on the first migrated data and the second migrated data, the migrated data is obtained.

[0109] The current batch of data can refer to the existing data in the current source mount point.

[0110] The method of copying the current batch of data to be migrated to the target mount point of the second sub-cluster may include: copying and pasting the current batch of data to the target mount point of the second sub-cluster.

[0111] Methods for adjusting the source mount point status to a non-writable state may include: adjusting the configuration data of the source mount point, changing the source mount point status from a read-write state to a non-writable state. The purpose of adjusting the source mount point status is because it is necessary to copy all the data mounted on the source mount point, so it is necessary to set the source mount point to a non-writable state before copying to ensure that no data is missed.

[0112] Methods for determining the remaining data of the data to be migrated may include: determining the remaining data of the data to be migrated through distcp's diff / rdiff / append.

[0113] Methods for copying the remaining data of the data to be migrated to the target mount point of the second sub-cluster may include: incremental copying based on the new changes in the data copied in the first copy using the diff / rdiff / append / update functions of distcp.

[0114] The method for obtaining migrated data based on the first migrated data and the second migrated data may include: adding the first migrated data and the second migrated data together to obtain the migrated data.

[0115] By migrating data in pre-defined batches, the source mount point can still be written to during the first copy, avoiding impact on business operations. After the first copy is completed, the remaining data is copied. At this time, the amount of remaining data is much smaller than the amount of data copied in the first copy, and the copying time is also shortened accordingly. Therefore, the duration of setting the source mount point to a non-writable state can be further shortened, further reducing the impact of data migration on business operations.

[0116] In this embodiment of the application, deleting the data to be migrated in the first sub-cluster includes:

[0117] Determine the mount table corresponding to the data to be migrated;

[0118] The mount table corresponding to the data to be migrated will be updated with the source mount point as the target mount point;

[0119] Set the wait time for deleting data to be migrated;

[0120] Based on the waiting time for deleting the data to be migrated, the data to be migrated is deleted from the first sub-cluster.

[0121] The mount table can refer to the mount table in the HDFS RBF federated cluster. This mount table is used to configure the data write path of the router. The router is the router in the HDFS RBF federated cluster. Users connect to the router through a client. The router determines the mount point path of the file through the mount table and performs read and write operations on the mount point path.

[0122] The waiting time for deleting data to be migrated can be a countdown. Once the countdown is up, the data to be migrated will be deleted. Although the source mount point cannot be written to, it can be read from. Other applications need to read data from the source mount point. Therefore, setting a waiting time for deleting data to be migrated allows other applications to read data from the source mount point normally, avoiding impact on business operations.

[0123] The method of updating the source mount point to the target mount point may include: updating the source mount point information to the target mount point information in the mount table, so that the router stores the newly arrived data to the target mount point through the updated mount table.

[0124] Depending on the waiting time for deleting the data to be migrated, the method for deleting the data to be migrated from the first sub-cluster may include: setting a waiting time for deleting the data to be migrated, and then using the cron service in Hadoop (distributed system infrastructure) to delete the data to be migrated after the specified time has elapsed.

[0125] In this embodiment of the application, after determining the source mount point of the first sub-cluster, the data to be migrated from the source mount point, and the target mount point of the second sub-cluster, the method further includes:

[0126] If the amount of data to be migrated is less than or equal to the preset amount of data to be migrated, the source mount point status will be adjusted to an unwritable state.

[0127] Based on the source mount point and the target mount point, the data to be migrated is copied to the target mount point of the second sub-cluster to obtain the migrated data.

[0128] In this embodiment of the application, it further includes:

[0129] When the data density difference of the first sub-cluster is less than the first threshold, stop migrating data from the first sub-cluster to the second sub-cluster.

[0130] When the difference in cluster data density between the second sub-cluster and the third threshold is greater than the third threshold, the data in the first sub-cluster is migrated to the third sub-cluster. The difference in sub-cluster data density between the third sub-cluster and the third threshold is greater than the fourth threshold.

[0131] In this embodiment of the application, it further includes:

[0132] According to the preset reporting mechanism, the real-time execution status information of the data migration system is sent to the state storage. The real-time execution status information is used to characterize the execution status of the data migration method.

[0133] The receiving state memory generates operation instructions based on real-time execution status information. The operation instructions include a continue execution instruction indicating that the execution status of the data migration method is normal, and a stop execution instruction indicating that the execution status of the data migration method is abnormal.

[0134] Control the data migration system according to the operating instructions.

[0135] The reporting mechanism can refer to a heartbeat mechanism, which involves periodically sending a custom structure (heartbeat packet) to let the other party know that it is still alive, thus ensuring the validity of the connection. In this embodiment, the heartbeat packet of the heartbeat mechanism is real-time execution status information.

[0136] The StateStore, in Hadoop, is used to collect resource information from various processes distributed across the cluster. In this embodiment, the StateStore is used to collect information and subsequently monitor the data migration process.

[0137] Real-time execution status information can include execution success and execution failure. In each step of the data migration, the execution result of the step is sent to the status memory after the step is completed.

[0138] The "continue execution instruction" can refer to the instruction fed back to the data migration system after the real-time execution status information received by the status memory indicates that the execution was successful.

[0139] A stop execution instruction can be a command sent to the data migration system when the real-time execution status information received by the status memory indicates an execution failure, or when no real-time execution status information is received within a preset time.

[0140] This application provides a data migration method that determines the data density difference in each sub-cluster of a federated cluster. The data density difference of a sub-cluster is determined based on a target data density and the data density of the sub-cluster itself. The target data density is determined based on the total storage capacity and overall storage usage of the federated cluster, while the sub-cluster data density is determined based on the total storage capacity and storage usage of the sub-cluster itself. Based on the data density difference in each sub-cluster of the federated cluster, a first sub-cluster and a second sub-cluster are determined within the federated cluster. The data density difference of the first sub-cluster is greater than a first threshold, and the data density difference of the second sub-cluster is less than a second threshold. The first threshold is greater than the second threshold. The first sub-cluster... The method for migrating data to the second sub-cluster involves comparing the data density difference of each sub-cluster with a first threshold and a second threshold to confirm the data density status of each sub-cluster. This determines which sub-clusters need to be migrated out and which need to be migrated in. The data migration procedure is triggered when the data volume of a sub-cluster exceeds the first threshold, avoiding manual judgment and enabling timely adjustment of the amount of stored data between sub-clusters. This improves the overall performance of the federated cluster. Batch migration is used to migrate data exceeding the preset data volume, reducing the time that the source mount point cannot write to large amounts of data during the migration process, thereby reducing the impact of federated cluster data rebalancing on data read and write operations.

[0141] Figure 3 A flowchart illustrating another data migration method provided in this application is shown below. Figure 3 As shown, the method includes:

[0142] S301. Determine the uniformity of data distribution based on the data density of the sub-cluster to identify the sub-cluster to be migrated and the sub-cluster to be migrated into.

[0143] The uniformity of data distribution is determined by the following method:

[0144] Let the total storage capacity of the federated cluster be total_capacity, and the total storage usage of the federated cluster be used_capacity. Then, the ideal data density (ideal_storage) is:

[0145] ideal_storage=used_capacity / total_capacity

[0146] Let the sub-cluster storage utilization rate be sub_used_radio, and the sub-cluster data density be sub_data_density.

[0147] sub_data_density=|ideal_storage-sub_used_radio|

[0148] The cluster provides the configuration parameter rbf.balance.threshold.percent, which triggers the data migration process when the following conditions are met:

[0149] |ideal_storage-sub_data_density|>rbf.balance.threshold.percent.

[0150] S302. Based on the sub-cluster to be migrated and the sub-cluster to be migrated into, determine the source mount point and target NameService of the sub-cluster to be migrated into, as well as the target path of the sub-cluster to be migrated into.

[0151] The source mount point is the source path (mount point) from which the data migration is expected to take place.

[0152] The target NameService is the NameService of the sub-cluster that needs data migration;

[0153] The target path is the full path of the sub-cluster to be migrated.

[0154] S303. Create a pipeline process based on the source mount point, the target NameService, and the target path, and store the source mount point in the StateStore.

[0155] For a single source mount point, only one rebalance process is allowed to run. If multiple rebalance processes are started, all processes except the first one will automatically disconnect.

[0156] S304. Verify the source mount point according to the preset verification scheme.

[0157] The verification scheme can refer to verifying whether the source mount point exists, whether the source mount point has sub-mount points, and whether the source path and the target path are different sub-clusters.

[0158] S305. If the verification result is successful, then based on the source mount point, target NameService, and target path, migrate the data of the sub-cluster to be migrated to the sub-cluster to be migrated.

[0159] In this embodiment of the application, if the verification result is successful, the data of the sub-cluster to be migrated is migrated to the sub-cluster to be migrated into based on the source mount point, the target NameService, and the target path, including:

[0160] If the verification result is successful, then the data of the sub-cluster to be migrated will be copied to the sub-cluster to be migrated, based on the source mount point, the target NameService, and the target path.

[0161] Verify that the data at the source mount point and the target path are consistent;

[0162] If the data at the source mount point and the target path are consistent, then execute the command to close the pipe process.

[0163] In this embodiment of the application, if the verification result is successful, the data of the sub-cluster to be migrated is copied to the sub-cluster to be migrated into based on the source mount point, the target NameService, and the target path, including:

[0164] This step sets two options to ensure data consistency during data balancing, while also ensuring business awareness:

[0165] 1) Option 1 (NO_WRITE): This parameter can be used for paths with small amounts of data. Its function is to set the source mount point directory to read-only in the StateStore before the data migration begins, so as to ensure that no write operations will occur during the data migration process and to ensure data consistency.

[0166] 2) Option Two (COPY_TWICE): If a data directory contains a large amount of data, and a single data copy takes several hours or more, setting the source mount point directory to read-only will prevent normal business operations during the data migration. This option is introduced to solve this problem. The function of this parameter is to perform data migration in two steps. In the first step, the business can perform normal read and write operations on the source directory until the first data copy is completed. In the second step, the source directory is set to read-only in the StateStore, and the diff / rdiff / append / update functions based on distcp perform incremental copying based on the new changes in the data copied in the first step. Through the above two steps, the impact of data migration on the business is minimized.

[0167] Meanwhile, an expiration time is set for the read-only lock during data replication, and the expiration time of the read-only lock needs to be continuously updated during the rebalance process. This is to prevent the source path mount point from remaining in a read-only state due to the data balancing process crashing.

[0168] In this embodiment of the application, verifying whether the data of the source mount point and the target path are consistent includes:

[0169] When this step is executed, the table name data has been successfully copied from the source path to the target path, and the read-only lock on the source mount point is still not released. At this time, the diff function of distcp is used again to generate a snapshot difference report to check whether the data on the source mount point and the target path are consistent. If they are consistent, the read-only lock is released; otherwise, the diff / rdiff / append / update operation is repeated.

[0170] In this embodiment of the application, if the data of the source mount point and the target path are consistent, then an instruction to close the pipeline process is executed, including:

[0171] 1) Update mount point: Update the MountTable information in the StateStore, and update the source mount point to the target path; wait for all routers to finish synchronizing before proceeding to the next step;

[0172] 2) Delete data in the source directory: This step allows you to set a waiting time for deleting source data to prevent read operations from failing. After the set waiting time is reached, the data will be deleted. After setting the deletion waiting time, the data will be retained and the pipeline process will be set to DELETE PENDING. At the same time, the cron service will be started and the data will be deleted after the specified time is reached.

[0173] 3) Delete the mount information of the source mount point: After the data is deleted, delete the mount information of the source mount point in the StateStore.

[0174] In this embodiment of the application, the method further includes:

[0175] The execution status and result of each step are reported to the StateStore via heartbeats. If the StateStore receives a failure message or loses a heartbeat, it considers the data migration to have failed and starts the Rollback process to restore the data to its initial state.

[0176] This application provides another data migration method that determines the uniformity of data distribution by assessing the data density of the sub-clusters, identifying the sub-clusters to be migrated and the sub-clusters to be migrated into. Based on these sub-clusters, it determines the source mount point and target NameService of the sub-cluster to be migrated, as well as the target path of the sub-cluster to be migrated into. A pipeline process is created based on the source mount point, target NameService, and target path. The source mount point is stored in the StateStore. A preset verification scheme is used to verify the source mount point. If the verification passes, the data from the sub-cluster to be migrated is migrated to the sub-cluster to be migrated into based on the source mount point, target NameService, and target path. This method automatically triggers sub-cluster data migration, avoiding impact on the business operations of the sub-clusters and thus improving their performance.

[0177] Figure 4 This application provides a schematic diagram of a data migration device, which includes a first determining module 401. 、 Second determination module 402 、 以及 Migration module 403, wherein:

[0178] The first determining module 401 is used to determine the data density difference in each sub-cluster of the federated cluster. The data density difference of the sub-cluster is determined based on the target data density and the sub-cluster data density of the sub-cluster. The target data density is determined based on the total storage of the federated cluster and the total storage usage. The sub-cluster data density is determined based on the total storage of the sub-cluster and the sub-cluster storage usage of the sub-cluster.

[0179] The second determining module 402 is used to determine the first sub-cluster and the second sub-cluster in the federated cluster based on the data density difference of each sub-cluster in the federated cluster, wherein the data density difference of the first sub-cluster is greater than a first threshold, the data density difference of the second sub-cluster is less than a second threshold, and the first threshold is greater than the second threshold.

[0180] Migration module 403 is used to migrate data from the first sub-cluster to the second sub-cluster.

[0181] In this embodiment of the application, the first determining module 401 is further configured to:

[0182] The target data density is obtained by calculating the ratio between the total storage usage and the total storage volume.

[0183] The data density of a sub-cluster is obtained by calculating the ratio between the storage usage of the sub-cluster and the total storage of the sub-cluster.

[0184] The difference between the sub-cluster data density and the target data density is calculated to determine the data density difference in each sub-cluster within the federated cluster.

[0185] In this embodiment of the application, the migration module 403 is further configured to:

[0186] Determine the source mount point of the first sub-cluster, the data to be migrated from the source mount point, and the target mount point of the second sub-cluster;

[0187] If the amount of data to be migrated is greater than the preset amount of data to be migrated, then the data to be migrated will be migrated to the target mount point of the second sub-cluster according to the preset batch migration amount.

[0188] In this embodiment of the application, the migration module 403 is further configured to:

[0189] If the amount of data to be migrated is greater than the preset amount of data to be migrated, then the data to be migrated will be copied to the target mount point of the second sub-cluster according to the preset batch migration amount, wherein the source mount point is in an unwritable state.

[0190] When the migration of the data to be migrated is completed, determine the migrated data of the target mount point in the second sub-cluster;

[0191] Compare the data to be migrated with the data that has already been migrated to obtain the comparison results;

[0192] Based on the comparison results, the source mount point status is changed from non-writable to writable.

[0193] Delete the data to be migrated from the first sub-cluster.

[0194] In this embodiment of the application, the migration module 403 is further configured to:

[0195] Determine the current batch of data to be migrated;

[0196] Based on the source mount point and the target mount point, the current batch of data to be migrated is copied to the target mount point of the second sub-cluster to obtain the first migrated data;

[0197] After the current batch of data is copied, change the source mount point status to a non-writable state;

[0198] Determine the remaining data to be migrated;

[0199] Based on the source mount point and the target mount point, the remaining data of the data to be migrated is copied to the target mount point of the second sub-cluster to obtain the second migrated data;

[0200] Based on the first migrated data and the second migrated data, the migrated data is obtained.

[0201] In this embodiment of the application, the migration module 403 is further configured to:

[0202] Determine the mount table corresponding to the data to be migrated;

[0203] The mount table corresponding to the data to be migrated will be updated with the source mount point as the target mount point;

[0204] Set the wait time for deleting data to be migrated;

[0205] Based on the waiting time for deleting the data to be migrated, the data to be migrated is deleted from the first sub-cluster.

[0206] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 50 includes:

[0207] The electronic device 50 may include a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a communication component 503, and other components. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0208] In the specific implementation process, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to execute the data migration method described above.

[0209] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0210] In the above Figure 5 In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0211] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0212] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0213] In some embodiments, a computer program product is also provided, comprising a computer program or instructions that, when executed by a processor, implement the steps in any of the above-described data migration methods.

[0214] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0215] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0216] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the data migration methods provided in embodiments of this application.

[0217] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0218] According to one aspect of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium.

[0219] Since the instructions stored in the storage medium can execute the steps of any of the data migration methods provided in the embodiments of this application, the beneficial effects that any of the data migration methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0220] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0221] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data migration method, characterized in that, The method, applied to a data migration system, includes: The data density difference in each sub-cluster of the federated cluster is determined based on the target data density and the sub-cluster data density of the sub-cluster. The target data density is determined based on the total storage of the federated cluster and the total storage usage. The sub-cluster data density is determined based on the total sub-cluster storage of the sub-cluster and the sub-cluster storage usage. Based on the data density difference of each sub-cluster in the federated cluster, the first sub-cluster and the second sub-cluster in the federated cluster are determined, wherein the data density difference of the first sub-cluster is greater than a first threshold, the data density difference of the second sub-cluster is less than a second threshold, and the first threshold is greater than the second threshold. Migrate the data from the first sub-cluster to the second sub-cluster.

2. The method according to claim 1, characterized in that, Determining the data density difference in each sub-cluster within the federated cluster includes: The target data density is obtained by calculating the ratio between the total storage usage and the total storage volume. The data density of the sub-cluster is obtained by calculating the ratio between the storage usage of the sub-cluster and the total storage of the sub-cluster. The difference between the sub-cluster data density and the target data density is calculated to determine the data density difference in each sub-cluster of the federated cluster.

3. The method according to claim 1, characterized in that, The step of migrating data from the first sub-cluster to the second sub-cluster includes: Determine the source mount point of the first sub-cluster, the data to be migrated from the source mount point, and the target mount point of the second sub-cluster; If the amount of data to be migrated is greater than the preset amount of data to be migrated, then the data to be migrated will be migrated to the target mount point of the second sub-cluster according to the preset batch migration amount.

4. The method according to claim 3, characterized in that, If the amount of data to be migrated is greater than a preset migration data amount, then the data to be migrated is migrated to the target mount point of the second sub-cluster according to the preset batch migration data amount, including: If the amount of data to be migrated is greater than the preset amount of data to be migrated, then the data to be migrated is copied to the target mount point of the second sub-cluster according to the preset batch migration amount, wherein the state of the source mount point is unwritable. When the migration of the data to be migrated is completed, the migrated data of the target mount point in the second sub-cluster is determined; By comparing the data to be migrated with the data that has already been migrated, a comparison result is obtained; Based on the comparison results, the state of the source mount point is adjusted from an unwritable state to a writable state; Delete the data to be migrated from the first sub-cluster.

5. The method according to claim 4, characterized in that, If the amount of data to be migrated is greater than a preset migration data amount, then the data to be migrated is copied to the target mount point of the second sub-cluster according to the preset batch migration data amount, including: Determine the current batch of data to be migrated; Based on the source mount point and the target mount point, the current batch of data to be migrated is copied to the target mount point of the second sub-cluster to obtain the first migrated data; After the current batch of data is copied, the source mount point status is adjusted to a non-writable state; Determine the remaining data of the data to be migrated; Based on the source mount point and the target mount point, the remaining data of the data to be migrated is copied to the target mount point of the second sub-cluster to obtain the second migrated data; Based on the first migrated data and the second migrated data, the migrated data is obtained.

6. The method according to claim 4, characterized in that, Deleting the data to be migrated from the first sub-cluster includes: Determine the mount table corresponding to the data to be migrated; The mount table corresponding to the data to be migrated will be updated with the source mount point as the target mount point. Set the wait time for deleting data to be migrated; Based on the waiting time for deleting the data to be migrated, the data to be migrated is deleted from the first sub-cluster.

7. The method according to claim 3, characterized in that, After determining the source mount point of the first sub-cluster, the data to be migrated from the source mount point, and the target mount point of the second sub-cluster, the method further includes: If the amount of data to be migrated is less than or equal to the preset amount of data to be migrated, then the state of the source mount point is adjusted to an unwritable state. Based on the source mount point and the target mount point, the data to be migrated is copied to the target mount point of the second sub-cluster to obtain the migrated data.

8. The method according to claim 1, characterized in that, The method further includes: When the data density difference of the first sub-cluster is less than the first threshold, the migration of data from the first sub-cluster to the second sub-cluster is stopped. When the cluster data density difference of the second sub-cluster is greater than the third threshold, the data in the first sub-cluster is migrated to the third sub-cluster. The sub-cluster data density difference of the third sub-cluster is less than the fourth threshold, and the third threshold is greater than the fourth threshold.

9. The method according to claim 1, characterized in that, The method further includes: According to a preset reporting mechanism, the real-time execution status information of the data migration system is sent to the state storage, and the real-time execution status information is used to characterize the execution status of the data migration method. The system receives an operation instruction generated by the state memory based on the real-time execution state information. The operation instruction includes a continue execution instruction indicating that the execution state of the data migration method is normal and a stop execution instruction indicating that the execution state of the data migration method is abnormal. Control the data migration system according to the operation instructions.

10. A data migration device, characterized in that, The device, used in a data migration system, includes: The first determining module is used to determine the data density difference in each sub-cluster of the federated cluster. The data density difference of the sub-cluster is determined based on the target data density and the sub-cluster data density of the sub-cluster. The target data density is determined based on the total storage of the federated cluster and the total storage usage. The sub-cluster data density is determined based on the total sub-cluster storage of the sub-cluster and the sub-cluster storage usage. The second determining module is used to determine the first sub-cluster and the second sub-cluster in the federated cluster based on the data density difference of each sub-cluster in the federated cluster, wherein the data density difference of the first sub-cluster is greater than a first threshold, the data density difference of the second sub-cluster is less than a second threshold, and the first threshold is greater than the second threshold. The migration module is used to migrate data from the first sub-cluster to the second sub-cluster.

11. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-9.

Images