Data migration storage method, device and management node
By detecting the reliability and loop structure of data blocks in a distributed cluster, the problem of data loss during data migration is solved, and the reasonable migration and reliability guarantee of data blocks are achieved.
Patent Information
- Application Number
- CN202210475723.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-29
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-04-29
AI Technical Summary
The prior art does not consider data reliability during data migration, which may lead to node exceptions and data loss, and lacks an effective data block migration mechanism.
In a distributed cluster, by obtaining the data blocks to be migrated and detecting whether they meet the preset reliability requirements, if they are satisfied, they will be migrated to the target storage node and disk. If they are not satisfied, they will be detected whether they form a loop structure. If they are formed, they will be marked as The migration is completed. If it is not formed, wait for the next migration operation.
It avoids subsequent abnormalities caused by data migration, ensures reasonable migration of data blocks, avoids unnecessary migration work, and ensures data integrity and reliability.
Smart Images

Figure CN114895849B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of distributed storage technology, and in particular to a data migration storage method, device, and management node. Background Art
[0002] As data expands, distributed storage systems have gradually become the mainstream of storage systems due to their scalability and high data reliability. Distributed block storage based on erasure codes has been widely used due to its advantages such as high space utilization.
[0003] The storage space created by the user can be split into multiple files (FILE). A file can be divided into multiple objects (OBJ) according to different redundancy strategies. Each OBJ consists of N+M data blocks (block). When the cluster encounters an abnormal situation, resulting in abnormal block data, the reliability of the cluster data will be reduced to a certain extent. At this time, data recovery will be triggered to ensure data reliability. The recovery of a large number of blocks caused by node offline also faces the problem of load imbalance after the node is restored, and recovery migration is required. When the cluster needs to expand nodes (disks) in the later stage, data migration of the cluster is also required. The reliability of data during the migration process has become a key consideration.
[0004] In the prior art, when performing data migration, the reliability requirements of the data during the migration process are not considered, which may result in abnormalities in the nodes during the migration, resulting in data loss. In addition, there is no complete mechanism in the prior art to ensure that each data block can be effectively and reasonably migrated. Summary of the invention
[0005] The purpose of the present application includes, for example, providing a data migration storage method, device and management node, which can avoid data loss and ensure that data blocks are reasonably migrated.
[0006] The embodiments of the present application can be implemented as follows:
[0007] In a first aspect, the present application provides a data migration storage method, which is applied to a management node in a distributed cluster, wherein the distributed cluster further includes a plurality of storage nodes connected to the management node, each of the storage nodes includes a plurality of disks, each of the disks is used to store data, and each data is divided into a plurality of data blocks, wherein the method includes:
[0008] When the migration operation is triggered, the data blocks to be migrated are obtained;
[0009] According to the data blocks in the destination storage node and the destination disk of the data blocks to be migrated, detecting whether the data blocks to be migrated meet the preset reliability requirements, and if the preset reliability requirements are met, migrating the data blocks to be migrated to the destination storage node and the destination disk;
[0010] If the preset reliability requirement is not met, detecting whether the data blocks to be migrated form a ring structure, wherein the ring structure is a structure in which the current storage nodes of two data blocks to be migrated are the destination storage nodes of each other;
[0011] If a ring structure is formed, the two data blocks to be migrated are directly marked as being in a migration completion state;
[0012] If a ring structure is not formed, wait for the next migration operation to be triggered.
[0013] In an optional implementation, the migration operation includes a recovery migration operation after the storage node or disk is offline and then online, and the method further includes a step of recovering the data block when the storage node or disk is offline, the step including:
[0014] When it is determined that the storage node or disk is offline, the data information of the data blocks on the offline storage node or disk is obtained and stored in the recovery table;
[0015] Scan the data information in the recovery table, and detect whether there are other data blocks belonging to the same data as the data block corresponding to the scanned data information in the recovery state;
[0016] If it exists, skip the data information and perform scanning of the data information of the next data block;
[0017] If it does not exist, the data block corresponding to the data information is sent to a destination storage node or a destination disk determined from other online storage nodes for recovery.
[0018] In an optional implementation manner, the step of determining that a storage node or a disk is offline includes:
[0019] When it is detected that a storage node or a disk is offline and the offline time exceeds a preset time, it is determined that the storage node or the disk is offline.
[0020] In an optional implementation manner, the migration operation includes a recovery migration operation after the storage node or disk is offline and then online again, and the step of detecting whether the data block to be migrated meets a preset reliability requirement based on the destination storage node and the data block in the destination disk of the data block to be migrated includes:
[0021] Detect whether there is a data block with the same data as the data block to be migrated currently in a waiting migration state; if not, detect whether there is a data block with the same data as the data block to be migrated in the destination disk;
[0022] If not, it is detected whether the number of data blocks of the same data as the data block to be migrated in the destination storage node is less than a preset maximum allowed number. If so, it is determined that the data block to be migrated meets the preset reliability requirement.
[0023] In an optional implementation, the migration operation includes capacity expansion migration when a new storage node is added in the distributed cluster, and before the step of triggering the migration operation, the method further includes:
[0024] Obtaining the used capacity of each storage node currently existing in the distributed cluster;
[0025] Based on the number of currently existing storage nodes, the used capacity of each currently existing storage node and the preset storage node capacity difference threshold, it is determined whether to perform capacity expansion migration between storage nodes. If it is determined to perform capacity expansion migration between storage nodes, the migration operation is triggered.
[0026] In an optional implementation, the step of determining whether to perform capacity expansion migration between storage nodes according to the number of currently existing storage nodes, the currently used capacity of each storage node, and a preset capacity difference threshold includes:
[0027] The average used capacity of the storage nodes is calculated based on the total used capacity of all storage nodes in the distributed cluster and the number of currently existing storage nodes;
[0028] For any storage node, based on the difference between the used capacity of the storage node and the average used capacity of the storage node, it is determined whether to perform capacity expansion migration between storage nodes.
[0029] In an optional implementation manner, the step of obtaining the data block to be migrated includes:
[0030] Comparing the used capacity of each storage node currently existing in the distributed cluster;
[0031] The storage node with the highest used capacity is used as the migration-out storage node, and the storage node with the lowest used capacity is used as the destination storage node;
[0032] A data block is extracted from the storage node to be migrated as the data block to be migrated.
[0033] In an optional implementation, the migration operation includes capacity expansion migration when a new storage node is added in the distributed cluster, and before the step of triggering the migration operation, the method further includes:
[0034] For a storage node in the distributed cluster where a newly added disk exists, obtaining the used capacity of each disk under the storage node;
[0035] Whether to perform capacity expansion migration between disks is determined based on the number of disks under the storage node, the used capacity of each disk, and a preset disk capacity difference threshold. If it is determined that capacity expansion migration between disks is to be performed, the migration operation is triggered.
[0036] In a second aspect, the present application provides a data migration storage device, which is applied to a management node in a distributed cluster, wherein the distributed cluster further includes a plurality of storage nodes connected to the management node, each of the storage nodes includes a plurality of disks, each of the disks is used to store data, each of the data is divided into a plurality of data blocks, and the device includes:
[0037] The acquisition module is used to acquire the data blocks to be migrated when the migration operation is triggered;
[0038] A first detection module, configured to detect whether the data block to be migrated meets a preset reliability requirement according to the destination storage node of the data block to be migrated and the data block in the destination disk;
[0039] A migration module, configured to migrate the data block to be migrated to the destination storage node and the destination disk when the preset reliability requirement is met;
[0040] A second detection module is used to detect whether the data blocks to be migrated form a ring structure when the preset reliability requirement is not met, wherein the ring structure is a structure in which the current storage nodes of two data blocks to be migrated are the destination storage nodes of each other;
[0041] A marking module, used for directly marking the two data blocks to be migrated as being in a migration completion state when forming a ring structure;
[0042] The waiting module is used to wait for the triggering of the next migration operation when a ring structure is not formed.
[0043] In a third aspect, the present application provides a management node, comprising one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine-executable instructions executable by the processor, and when the management node is running, the processor executes the machine-executable instructions to perform the method steps described in any one of the aforementioned embodiments.
[0044] The beneficial effects of the embodiments of the present application include, for example:
[0045] The present application provides a data migration storage method, device and management node. When a migration operation is triggered, the data blocks to be migrated are obtained. According to the data blocks in the destination storage node and the destination disk of the data blocks to be migrated, when it is determined that the data blocks to be migrated meet the preset reliability requirements, the data blocks to be migrated are migrated to the destination storage node and the destination disk. When the preset reliability requirements are not met, it is detected whether the data blocks to be migrated form a ring structure. If so, the data blocks to be migrated are directly marked as migration completed. If not, the next migration operation is waited for to be triggered. In this scheme, it is determined whether the data blocks to be migrated meet the preset reliability requirements based on the data blocks in the target storage node and the destination disk, so as to avoid the problem of subsequent anomalies and data loss caused by the migration. In addition, for the data blocks to be migrated that form a special ring structure, the method of directly marking the migration successfully is adopted, which can avoid unnecessary migration work and ensure that each data block can be reasonably migrated. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0047] Figure 1 A schematic diagram of the structure of a distributed cluster provided in an embodiment of the present application;
[0048] Figure 2 A flowchart of a data migration and storage method provided in an embodiment of the present application;
[0049] Figure 3 A flowchart of a data recovery method in a data migration and storage method provided in an embodiment of the present application;
[0050] Figure 4 for Figure 2 Flow chart of sub-steps included in step S202;
[0051] Figure 5 A schematic diagram of a data block in a ring structure provided in an embodiment of the present application;
[0052] Figure 6 A schematic diagram of the original distribution of data blocks in the same group of data provided in an embodiment of the present application;
[0053] Figure 7 A schematic diagram of the distribution of data blocks in the same group of data after multiple recovery provided in an embodiment of the present application;
[0054] Figure 8 A schematic diagram of data migration and storage processing provided in an embodiment of the present application;
[0055] Fig. 9 One of the flow charts of the method for determining capacity expansion and migration in the data migration and storage method provided in the embodiment of the present application;
[0056] Fig.10 The second flowchart of the method for determining capacity expansion and migration in the data migration and storage method provided in the embodiment of the present application;
[0057] Fig.11 for Figure 2 Flow chart of sub-steps included in step S201;
[0058] Fig.12 This is a diagram showing the used capacity of each node when a new node is added to the distributed cluster;
[0059] Fig.13 It is the specific value of the used capacity of each node when adding a new node in the distributed cluster;
[0060] Fig.14 A schematic diagram showing the used capacity of each node when adding new nodes and performing migration in a distributed cluster;
[0061] Fig.15 The specific value of the used capacity of each node after adding new nodes and migrating them to the distributed cluster;
[0062] Fig.16 A structural block diagram of a management node provided in an embodiment of the present application;
[0063] Fig.17 This is a functional module block diagram of the data migration storage device provided in an embodiment of the present application.
[0064] Icons: 110 - storage medium; 120 - processor; 130 - data migration storage device; 131 - acquisition module; 132 - first detection module; 133 - migration module; 134 - second detection module; 135 - marking module; 136 - waiting module; 140 - communication interface. DETAILED DESCRIPTION
[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.
[0066] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0067] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0068] In the description of the present application, it should be noted that the terms "first", "second", etc., if used, are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0069] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.
[0070] See also Figure 1 , is a schematic diagram of the structure of a distributed cluster provided in an embodiment of the present application. The distributed cluster includes a management node and multiple storage nodes (data nodes) connected to the management node. Each storage node includes multiple disks. The management node and the storage node can be a terminal device or a server. The management node and the storage node form an operating framework of the storage system.
[0071] Each disk can be used to store data, and each data is divided into multiple data blocks, where data blocks belonging to the same data can be divided into the same group. For example, the created storage space can be divided into multiple files (FILE), and a file can be divided into multiple objects (OBJ) according to different redundancy strategies. Each OBJ consists of N+M data blocks (blocks).
[0072] See also Figure 2 , is a flow chart of a data migration and storage method provided in an embodiment of the present application, and the migration and storage method can be applied to the management node in the above-mentioned distributed cluster. Figure 2 As shown, the data migration storage method provided in the embodiment of the present application is specifically described.
[0073] S201, when a migration operation is triggered, obtaining a data block to be migrated.
[0074] S202, based on the destination storage node of the data block to be migrated and the data block in the destination disk, detect whether the data block to be migrated meets the preset reliability requirement, if it meets the preset reliability requirement, execute the following step S203, if it does not meet the preset reliability requirement, execute the following step S204.
[0075] S203: Migrate the data block to be migrated to the destination storage node and destination disk.
[0076] S204, detect whether the data blocks to be migrated form a ring structure, wherein the ring structure is a structure in which the current storage nodes of the two data blocks to be migrated are each other's destination storage nodes. If a ring structure is formed, execute the following step S205; if a ring structure is not formed, execute the following step S206.
[0077] S205: directly mark the two data blocks to be migrated as being in a migration completion state.
[0078] S206, waiting for the next migration operation to be triggered.
[0079] In this embodiment, when a storage node or disk in a distributed cluster is abnormal, the reliability of the cluster data will be affected, and the data on the abnormal storage node or disk needs to be restored. After the abnormal storage node or disk is online again, the previously restored data needs to be migrated to the original storage node or disk to ensure load balancing within the cluster. In addition, when there are new storage nodes or disks in the cluster, there will also be a problem of unbalanced load within the cluster, which will also trigger the migration operation.
[0080] When the migration operation is triggered, the data blocks to be migrated can be obtained. When the abnormal storage node or disk is online again, the data blocks to be migrated can be the data blocks that were originally on the abnormal storage node or disk and have now been restored to other storage nodes. In the case of adding a new storage node or disk, the data blocks to be migrated are the data blocks on the storage nodes that originally existed in the distributed cluster.
[0081] When the abnormal storage node or disk comes online again, the destination storage node and destination disk of the data blocks to be migrated are the storage node or disk that comes online again. When a new storage node or disk is added to the cluster, the destination storage node and destination disk of the data blocks to be migrated may be the new storage node and the new disk.
[0082] After determining the data blocks to be migrated and the destination storage node and destination disk, the data blocks on the destination storage node and destination disk can be obtained, and combined with the data blocks to be migrated to determine whether the data blocks to be migrated meet the preset reliability requirements, that is, whether the data blocks to be migrated are currently suitable for migration to the destination storage node and destination disk. If it is determined that the data blocks to be migrated meet the preset reliability requirements, the data blocks to be migrated can be directly migrated to the destination storage node and destination disk.
[0083] If it is determined that the data block to be migrated does not meet the preset reliability requirements, in general, it is necessary to wait for the next migration operation to be triggered before determining whether to migrate. However, there are some special cases. For example, the data blocks to be migrated constitute a ring structure, that is, the current data blocks to be migrated include at least two, and the current storage nodes of the two data blocks to be migrated are each other's destination storage nodes. For example, the data blocks to be migrated include data block a and data block b, the current storage node of data block a is storage node A and the destination storage node is storage node B, and the current storage node of data block b is storage node B and the destination storage node is storage node A. That is, it is necessary to migrate data block a from storage node A to storage node B, and migrate data block b from storage node B to storage node A.
[0084] When the data blocks to be migrated do not meet the preset reliability requirements, but form the above-mentioned ring structure, the two data blocks to be migrated can be directly marked as migration completed, that is, no actual migration action is required. If the data blocks to be migrated neither meet the preset reliability requirements nor form the above-mentioned ring structure, it is necessary to wait for the next migration operation to be triggered before making a migration judgment.
[0085] The data migration storage method provided in this embodiment determines whether the data blocks to be migrated meet the preset reliability requirements based on the data blocks in the target storage node and the destination disk, thereby avoiding the problem of data loss caused by subsequent abnormalities that may be caused by migration. In addition, for the data blocks to be migrated that form a special ring structure, a method of directly marking the migration as successful is adopted, which can avoid unnecessary migration work and ensure that each data block can be reasonably migrated.
[0086] In this embodiment, the migration operation includes a recovery migration operation after a storage node or disk is offline and then online, and a capacity expansion migration when a new storage node or a new disk exists in the distributed cluster.
[0087] When the migration operation is a recovery migration operation after the storage node or disk is offline and then online, it means that the storage node or disk was offline before. To ensure that data is not lost, the data blocks are recovered when the storage node or disk is offline. Therefore, please refer to Figure 3 The data migration storage method provided in this embodiment also includes the following step of recovering the data block when the storage node or disk is offline.
[0088] S101, when it is determined that a storage node or a disk is offline, data information of a data block on the offline storage node or disk is obtained and stored in a recovery table.
[0089] S102, scan the data information in the recovery table, and detect whether there are other data blocks belonging to the same data as the data block corresponding to the scanned data information in the recovery state. If so, execute the following step S103, if not, execute the following step S104.
[0090] S103, skipping the data information and performing scanning of the data information of the next data block;
[0091] S104: Send the data block corresponding to the data information to a destination storage node or a destination disk determined from other online storage nodes for recovery.
[0092] In this embodiment, when a storage node or disk is offline for more than a specified time, the data blocks on it will be deemed unavailable. Therefore, the data blocks on the offline storage node or disk will be used as data blocks to be recovered, and the data information of the data blocks on the offline storage node or disk can be stored in the recovery table.
[0093] In order to avoid the storage node or disk being offline for a short time, such as unplugging and plugging the disk, node restarting, etc., which triggers the recovery operation and wastes cluster resources, therefore, in this embodiment, when the storage node or disk is detected to be offline and the offline duration exceeds the preset duration, the storage node or disk is determined to be offline, and then the data block recovery operation is triggered. In this way, the problem of wasting resources due to data block recovery caused by a short offline time can be avoided.
[0094] In addition, in this embodiment, when it is detected that the storage node or disk is offline and the offline time does not exceed the preset time, but there is a read and write operation on the data block therein, the read and write operation will fail. At this time, although the storage node or disk is not determined to be offline, the data block pointed to by the read and write operation still needs to be restored. Similarly, the data information of the data block pointed to by the read and write operation can be stored in the recovery table to wait for recovery. In this way, the read and write operation can be avoided from being affected.
[0095] In this embodiment, the data information of the data blocks on the offline storage node or disk and the data information of the data blocks pointed to and failed by the read and write operations are stored in the recovery table shown below to wait for recovery, and the status of the corresponding data blocks is updated to the offline status.
[0096] Table 1 Recovery table
[0097] describe Table name Type Key Value Recovery Table TBL_RECOVER Hash FILEID-OBJID-BLOCKID timestamp-state
[0098] Among them, state indicates the recovery status, 0 means recovery has not started, 1 means recovery is ready, and 2 means recovery is in progress.
[0099] The recovery thread periodically scans the data information in the recovery table. The number of data information that can be scanned each time can be a specified number. For the scanned data information, it can first be detected whether there are other data blocks of the same data that are in the recovery state, that is, whether there are data blocks in the same group that are in the recovery. If so, the data information is skipped and waits for the next scan. If not, the data block corresponding to the data information can be recovered, and the destination storage node can be determined from the online storage nodes in the current distributed cluster. If it is a recovery between disks, the destination disk can be determined from it. The data block corresponding to the scanned data information can be recovered to the destination storage node or destination disk.
[0100] After the data block is restored, the management node can receive the reported recovery confirmation message. If the recovery status in the reported recovery confirmation message is normal, it can be determined that the data block has been successfully restored. Otherwise, the data block recovery has failed and needs to be restored again. After the data block is successfully restored, the data information of the data block can be deleted from the recovery table and stored in the recovery completion table shown below. The data block corresponding to the data information in the recovery completion table is waiting for the migration operation.
[0101] Table 2 Recovery completion table
[0102] describe Table name Type Key Recovery completion table TBL_RECOVERD-Source DNID-Source DISKID Set FILEID-OBJID-BLOCKID
[0103] Among them, the source DNID and the source DISKID record the initial position of the data block.
[0104] After all data blocks corresponding to the data information in the recovery table are restored, that is, after all the data information in the recovery table is stored in the recovery completion table. For the data information in the recovery completion table, the migration scanning thread can periodically scan the data information therefrom, and if it is determined that the corresponding data block meets the preset reliability requirements, it is stored in the migration table shown below, waiting for data migration. For the data information in the recovery completion table, if the data to which the data block corresponding to the data information belongs does not exist in the database, the data information can be deleted from the recovery completion table. In addition, if the destination disk of the data block corresponding to the data information is not online, the data information can be skipped to wait for the next migration scan.
[0105] Table 3 Migration table
[0106]
[0107] Among them, SrcDNID and SrcDISKID record the location of the current storage node and disk of the data block, DesDNID and DesDISKID record the location of the destination storage node and disk, RecvedDataType indicates the received data type (1-recovery migration type; 2-capacity expansion migration type), MgrStatus indicates the migration status of the data block (0-migration has not started; 1-waiting for migration; 2-migrating; 3-migration ended), and Timestamp indicates the time of entering the migration table.
[0108] See also Figure 4 In this embodiment, when determining whether a data block meets the reliability requirement, it can be implemented in the following manner:
[0109] S2021, detecting whether there is a data block that has the same data as the data block to be migrated and is in a waiting state for migration. If not, executing the following step S2022; if so, executing the following step S2025.
[0110] S2022, detect whether there is a data block in the destination disk that has the same data as the data block to be migrated. If not, execute the following step S2023; if so, execute the following step S2025.
[0111] S2023, then detect whether the number of data blocks in the destination storage node that contain the same data as the data block to be migrated is less than the preset maximum allowed number, if so, execute the following step S2024, if not, execute the following step S2025.
[0112] S2024: Determine whether the data block to be migrated meets a preset reliability requirement.
[0113] S2025: Determine whether the data block to be migrated does not meet a preset reliability requirement.
[0114] As can be seen from the above, the migration scanning thread can scan data information from the recovery completion table and store it in the migration table. After scanning the data information, it can first be detected whether there is a data block that belongs to the same data as the data block corresponding to the data information and is in a waiting state for migration, that is, whether there is data information of the data block belonging to the same data in the migration table. If so, it is determined that the preset reliability requirement is not met. If not, subsequent detection actions can be performed.
[0115] In addition, it is possible to detect whether there are data blocks on the destination disk that have the same data as the data blocks to be migrated, that is, whether there are data blocks in the same group. If so, it can be determined that the preset reliability requirement is not met. If not, it is also possible to detect the size relationship between the number of data blocks on the destination storage node that have the same data as the data blocks to be migrated and the preset maximum allowable number. The preset maximum allowable number can be calculated by the following method:
[0116] MaxNum=HighAlign(N+M,DnNum) / DnNum
[0117] Among them, N and M respectively represent the number of data blocks into which the data in the erasure code is divided and the number of redundant fragments generated according to a certain algorithm, HighAlign means upward alignment, and DnNum represents the number of storage nodes in the distributed cluster.
[0118] If the number of data blocks on the destination storage node that belong to the same data as the data block to be migrated is less than the preset maximum allowable number, it can be determined that the preset reliability requirement is met. If it is greater than or equal to the preset maximum allowable number, it can be determined that the preset reliability requirement is not met.
[0119] After determining that the preset reliability requirement is met in the above manner, the data information of the data block to be migrated can be stored in the migration table, and the migration status of the data block to be migrated can be updated from not started migration to waiting migration status.
[0120] The migration processing thread can obtain the data blocks waiting for migration and send them to the destination storage node and destination disk for migration. After the sending is successful, the migration status is updated to Migrating. If the sending fails, the migration status of the data block is updated to Migration Not Started and waits for the next migration scan. The next migration scan here refers to the scan of the data information already in the migration table.
[0121] In addition, during the migration process, if the data block is in the process of migration for more than the specified time, a timeout process can be performed, and the data block is rolled back and inserted into the recovery completion table, and deleted from the migration table to wait for the next migration scan. The next migration scan here refers to the scan of the data information in the recovery completion table.
[0122] If the above method determines that the preset reliability requirement is not met, but the data blocks to be migrated can form a ring structure, for example, as shown in 5, data block b1 is restored to Node2, and data block b2 is restored to Node1. At this time, due to reliability limitations, b1 and b2 cannot be migrated back to their respective initial positions. Similarly, a ring structure can also be formed when there are more than two data blocks.
[0123] In this case, b1 needs to be migrated back to Node1. At this time, you can query the data block that actually exists on Node1 (this time it is b2). Then determine whether b2 is the data block to be migrated back to Node2. If so, the location to be migrated of b1 can be modified to the location to be migrated of b2, and the migration request information of b2 can be deleted. Then determine whether the location to be migrated of b1 is the same as the current location of b1 (the current location is Node2). If they are the same, the migration request information of b1 can be deleted. That is, in this way, it can be determined whether b1 and b2 form a ring structure, and when forming a ring structure, their respective states can be modified to the migration completion state, that is, there is no need to perform actual migration operations.
[0124] If they are different, that is, they do not form a ring structure, then wait for the next migration operation to be triggered.
[0125] Furthermore, combined with Figure 6 and Figure 7 As shown, the migration of the ring structure is illustrated as an example:
[0126] In a 4-node cluster with an erasure ratio of 4+2, node 1 is offline, and the data blocks of node 1 are restored to the other three nodes. Then node 2 is offline, and node 1 is online. After the recovery, the disk is offline again. Since recovery takes precedence over migration, after multiple recovery, the migration of data blocks in the same group of files will form a loop. Take file 67444838135472 as an example. The changes of the 6 blocks in the same group are as follows: Figure 6 , 7 As shown in (fileid-objid-blockid).
[0127] After multiple recoveries, 67444838135472-1-6 was recovered to node 2 disk 3, and 67444838135472-1-3 was recovered to node 1 disk 1. The information in the recovery completion table is: "TBL_RECOVERD-1-1", "67444838135472-1-6", "TBL_RECOVERD-2-3", "67444838135472-1-3". The two data blocks need to be migrated to each other's nodes, but the migration judgment no longer meets the preset reliability requirements. When a ring is formed, the migration position of 67444838135472-1-6 is changed to the position to be migrated of 67444838135472-1-3, and the information to be migrated of 67444838135472-1-3 is deleted, that is, the information in the recovery completion table is changed to: "TBL_RECOVERD-2-3" "67444838135472-1-6". The next migration judgment is performed, and the migration position is the same as the current position, so there is no need to migrate. It is deleted from the recovery completion table, and the migration of the ring data block is completed.
[0128] After the migration is completed, the management node can receive the reported migration results and confirm whether the migration is successful through the migration status in the migration results. If the migration is successful, the current position of the data block in the data can be updated and its data information can be deleted from the migration table. Otherwise, the migration needs to be repeated.
[0129] After all data blocks corresponding to the data information in the recovery completion table have been migrated, that is, when the recovery completion table is empty, the recovery migration operation is completed.
[0130] As can be seen from the above, in this embodiment, the migration operation also includes capacity expansion migration when there are new storage nodes or new disks in the distributed cluster. Figure 8 As shown in , the recovery module can be used to recover the data block, and the recovery migration module can be used to implement the recovery migration process. In addition, the expansion migration module can be used to implement the expansion migration process.
[0131] When adding a new storage node or a new disk, you need to first determine whether the nodes in the cluster are balanced, or whether the disks in the node are balanced. If they are unbalanced, you need to migrate the data blocks.
[0132] In this embodiment, it can be seen from the above that the migration table contains data information of data blocks to be migrated, and the number of data information in the migration table can be periodically scanned. If the number of data information in the migration table is greater than a preset threshold, it indicates that there are many data blocks to be migrated at present, and the expansion migration can be temporarily not triggered, waiting for the next round of scanning. If the number of data information in the migration table is less than or equal to the preset threshold, it can be determined whether the nodes or disks are balanced. If not, the expansion migration is triggered.
[0133] See also Fig. 9 In this embodiment, when checking whether the storage nodes are balanced, before triggering the migration operation, the following method can be used to check whether the cluster is balanced:
[0134] S301, obtaining the used capacity of each storage node currently existing in the distributed cluster.
[0135] S302, judging whether to perform capacity expansion migration between storage nodes based on the number of currently existing storage nodes, the used capacity of each currently existing storage node and a preset storage node capacity difference threshold, and if it is determined to perform capacity expansion migration between storage nodes, triggering the migration operation.
[0136] In this embodiment, the storage nodes currently existing in the distributed cluster include the storage nodes originally existing in the cluster and the newly added storage nodes.
[0137] In this embodiment, in addition to the newly added storage nodes in the cluster, the used capacity of each storage node in the originally existing storage nodes may be the same or different. The total used capacity of the storage nodes of all storage nodes in the distributed cluster can be calculated, that is, the used capacity of all storage nodes is added up. The average used capacity of the storage nodes is calculated based on the total used capacity of the storage nodes of all storage nodes in the distributed cluster and the number of currently existing storage nodes. Then, for any storage node, based on the difference between the used capacity of the storage node and the average used capacity of the storage node, it is determined whether to perform capacity expansion migration between storage nodes.
[0138] Specifically, when the absolute value of the difference between the used capacity of any storage node and the average used capacity of the storage nodes is greater than a preset storage node capacity difference threshold, it can be determined that the cluster is unbalanced and capacity expansion and migration are required.
[0139] In this embodiment, when the following formula is satisfied, it can be determined that capacity expansion migration is required:
[0140] Labs(nodeUsedCap-(nodeTotalUsedCap / nodeNum))>nodeUsedCapThread
[0141] Where labs represents the absolute value, nodeUsedCap represents the used capacity of any storage node, nodeTotalUsedCap represents the total used capacity of the storage nodes, nodeNum represents the number of storage nodes, and nodeUsedCapThread represents the preset storage node capacity difference threshold.
[0142] In addition, if you add a new disk to a storage node, you can also first determine whether the node is unbalanced. Fig.10 In this embodiment, it is also possible to determine in advance whether to perform capacity expansion migration between disks under the node by the following methods:
[0143] S401, for a storage node having a newly added disk in the distributed cluster, obtain the used capacity of each disk under the storage node.
[0144] S402, judging whether to perform capacity expansion migration between disks according to the number of disks under the storage node, the used capacity of each disk and a preset disk capacity difference threshold, and if it is determined to perform capacity expansion migration between disks, triggering the migration operation.
[0145] In this embodiment, when a new disk is added under the storage node, it can be determined whether to migrate based on the usage of the current disk. The total used capacity of all disks can be calculated, and the average used capacity of the disks can be calculated based on the total number of disks. For any disk, the difference between the used capacity of the disk and the average used capacity of the disk can be calculated, and whether to migrate can be determined based on the size relationship between the difference and the preset disk capacity difference threshold. Specifically, when the absolute value of the difference is greater than the disk capacity difference threshold, it can be determined that migration is required.
[0146] In this embodiment, whether to perform migration between disks can be determined by the following formula:
[0147] Labs(diskUsedCap-(nodeUsedCap / diskNum))>diskUsedCapThread
[0148] Among them, diskUsedCap represents the used capacity of any disk, nodeUsedCap is the used capacity of the node (that is, the total used capacity of the disk), diskNum is the number of disks in the storage node, and diskUsedCapThread is the preset capacity difference threshold between disks.
[0149] In this embodiment, if the cluster load is balanced or the node load is balanced, the next scan judgment can be performed after waiting for a certain period of time. If it is determined through the above method that data migration between nodes or data migration between disks is required, the migration operation can be performed until the cluster or node load is balanced. Fig.11 , when performing a migration operation, the acquired data blocks to be migrated may be determined in the following manner:
[0150] S2011, comparing the used capacities of the respective storage nodes currently existing in the distributed cluster.
[0151] S2012: Use the storage node with the highest used capacity as the migration-out storage node, and use the storage node with the lowest used capacity as the destination storage node.
[0152] S2013: extracting data blocks from the storage node to be migrated as data blocks to be migrated.
[0153] In this embodiment, when the load in the cluster is unbalanced, the storage node to be migrated can be determined based on the high-low ranking structure of the used capacity of each storage node. Similarly, the disk to be migrated can also be determined. The destination storage node and destination disk are determined. The principle of migration is to migrate from the node or disk with the highest used capacity to the storage node or disk with the lowest used capacity.
[0154] Therefore, the data blocks to be migrated are extracted from the migrated storage node or the migrated disk. A specified number of data blocks can be extracted from the migrated storage node or the migrated disk, and the data information of the data blocks that meet the reliability requirements are stored in the migration table waiting for migration. After the migration is successful, the data information of the data block can be deleted from the migration table. If the migration fails, the data information of the data block is also deleted from the migration table, but there is no need to roll back to the recovery completion table and wait for the next round of expansion migration scanning.
[0155] The following example illustrates the capacity expansion and migration process when a new storage node is added to the cluster:
[0156] In a three-node cluster, each node is equipped with 10 8TB disks, the erasure ratio is 4+2, and multiple files are created. The total capacity occupies 5000GB. The used capacity of each node is 2500GB, and the used capacity of each disk is 250GB. After the file is created, a new node 4 with the same configuration is expanded into the cluster. After the expansion is successful, the used capacity of node 4 is 0, triggering the expansion migration. Fig.12 as shown in .
[0157] During capacity expansion and migration, you can poll each node and disk to obtain the used capacity, and sort them according to the used capacity. The sorting results are as follows: Fig.13 as shown in .
[0158] Then determine whether the expansion migration conditions are met, set the expansion migration capacity difference threshold to 500GB, and calculate the capacity difference: lab(2500GB–(2500*3)GB / 4)=625GB>threshold 500GB, which can determine that the expansion migration conditions are met. According to the above sorting results, select DN1 and Disk 1 with the largest used capacity after sorting as the migration-out DN and migration-out disk, and DN4 as the migration-in node. Obtain the data information of 100 blocks on DN1-DISK1 that meet the migration reliability and put them into the migration table TBL_MIGRATION waiting for migration.
[0159] When the number of migration tables is less than 10, the used capacity of each DN and disk is obtained again to calculate the capacity for expansion and migration. If each node completes a group of 100 block migrations, nodes 1 / 2 / 3 have a capacity of 64M*100=6400M=6.25GB to migrate to node 4, where 64M refers to the size of a data block. At this time, the used capacity of each node and disk is as follows: Fig.14 as shown in .
[0160] You can poll to obtain the used capacity of each node and disk, and sort them according to the used capacity. The sorting results are as follows: Fig.15 as shown in .
[0161] Calculate the capacity difference: lab(2493.75GB–(2493.75*3+18.75)GB / 4)=618.75GB>threshold 500GB, which meets the capacity expansion migration conditions. Continue to perform capacity expansion migration according to the same steps as above until the capacity difference calculation result is no greater than the threshold 500GB. Then stop the capacity expansion migration.
[0162] The data migration and storage method provided in this embodiment can migrate the restored data blocks to the original location after the cluster returns to normal, thereby ensuring the reliability of the cluster data and the balance of the cluster capacity. In addition, for data blocks that form a ring structure but do not meet the preset reliability requirements and cannot be migrated back, the ring structure is destroyed and the data blocks are directly set to the migration completion state to ensure that all data blocks can be migrated normally. In addition, in the scenario where the cluster does not meet the capacity usage requirements and the expansion of new disks and nodes leads to unbalanced node capacity, the expansion migration can ensure that the cluster balances the capacity of each node and disk without affecting normal use, thereby ensuring cluster load balancing.
[0163] See also Fig.16 , which is an exemplary component diagram of a management node provided in an embodiment of the present application. The management node may include a storage medium 110, a processor 120, a data migration storage device 130, and a communication interface 140. In the present embodiment, the storage medium 110 and the processor 120 are both located in the management node and the two are separately arranged. However, it should be understood that the storage medium 110 may also be independent of the management node and may be accessed by the processor 120 through a bus interface. Alternatively, the storage medium 110 may also be integrated into the processor 120, for example, it may be a cache and / or a general register.
[0164] The data migration storage device 130 can be understood as the above-mentioned management node, or the processor 120 of the management node, and can also be understood as a software function module independent of the above-mentioned management node or processor 120 that implements the above-mentioned data migration storage method under the control of the management node.
[0165] like Fig.17 As shown, the data migration storage device 130 may include an acquisition module 131, a first detection module 132, a migration module 133, a second detection module 134, a marking module 135 and a waiting module 136. The functions of each functional module of the data migration storage device 130 are described in detail below.
[0166] The acquisition module 131 is used to acquire the data block to be migrated when the migration operation is triggered;
[0167] It can be understood that the acquisition module 131 can be used to execute the above step S201. The detailed implementation of the acquisition module 131 can refer to the above content related to step S201.
[0168] A first detection module 132, configured to detect whether the data block to be migrated meets a preset reliability requirement according to the destination storage node of the data block to be migrated and the data block in the destination disk;
[0169] It can be understood that the first detection module 132 can be used to execute the above step S202. The detailed implementation of the first detection module 132 can refer to the above content related to step S202.
[0170] A migration module 133, configured to migrate the data block to be migrated to the destination storage node and the destination disk when the preset reliability requirement is met;
[0171] It can be understood that the migration module 133 can be used to execute the above step S203. The detailed implementation of the migration module 133 can refer to the above content related to step S203.
[0172] A second detection module 134 is used to detect whether the data blocks to be migrated form a ring structure when the preset reliability requirement is not met, wherein the ring structure is a structure in which the current storage nodes of two data blocks to be migrated are the destination storage nodes of each other;
[0173] It can be understood that the second detection module 134 can be used to execute the above step S204. The detailed implementation of the second detection module 134 can refer to the above content related to step S204.
[0174] A marking module 135, used for directly marking the two to-be-migrated data blocks as being in a migration completion state when forming a ring structure;
[0175] It can be understood that the marking module 135 can be used to execute the above step S205. The detailed implementation of the marking module 135 can refer to the above content related to step S205.
[0176] The waiting module 136 is used to wait for the triggering of the next migration operation when a ring structure is not formed.
[0177] It can be understood that the waiting module 136 can be used to execute the above step S206. The detailed implementation of the waiting module 136 can refer to the above content related to step S206.
[0178] In a possible implementation, the migration operation includes a recovery migration operation after the storage node or disk is offline and then online. The data migration storage device 130 further includes a recovery module, which can be used to:
[0179] When it is determined that the storage node or disk is offline, the data information of the data blocks on the offline storage node or disk is obtained and stored in the recovery table;
[0180] Scan the data information in the recovery table, and detect whether there are other data blocks belonging to the same data as the data block corresponding to the scanned data information in the recovery state;
[0181] If it exists, skip the data information and perform scanning of the data information of the next data block;
[0182] If it does not exist, the data block corresponding to the data information is sent to a destination storage node or a destination disk determined from other online storage nodes for recovery.
[0183] In a possible implementation, the above recovery module may be used to:
[0184] When it is detected that a storage node or a disk is offline and the offline time exceeds a preset time, it is determined that the storage node or the disk is offline.
[0185] In a possible implementation, the migration operation includes a recovery migration operation after the storage node or disk is offline and then online again, and the first detection module 132 may be used to:
[0186] Detect whether there is a data block with the same data as the data block to be migrated currently in a waiting migration state; if not, detect whether there is a data block with the same data as the data block to be migrated in the destination disk;
[0187] If not, it is detected whether the number of data blocks of the same data as the data block to be migrated in the destination storage node is less than a preset maximum allowed number. If so, it is determined that the data block to be migrated meets the preset reliability requirement.
[0188] In a possible implementation, the migration operation includes capacity expansion migration when a new storage node is added in the distributed cluster, and the data migration storage device 130 may further include a judgment module, which may be used to:
[0189] Obtaining the used capacity of each storage node currently existing in the distributed cluster;
[0190] Based on the number of currently existing storage nodes, the used capacity of each currently existing storage node and the preset storage node capacity difference threshold, it is determined whether to perform capacity expansion migration between storage nodes. If it is determined to perform capacity expansion migration between storage nodes, the migration operation is triggered.
[0191] In a possible implementation manner, the above-mentioned determination module may be used to:
[0192] The average used capacity of the storage nodes is calculated based on the total used capacity of all storage nodes in the distributed cluster and the number of currently existing storage nodes;
[0193] For any storage node, based on the difference between the used capacity of the storage node and the average used capacity of the storage node, it is determined whether to perform capacity expansion migration between storage nodes.
[0194] In a possible implementation, the acquisition module 131 may be used to:
[0195] Comparing the used capacity of each storage node currently existing in the distributed cluster;
[0196] The storage node with the highest used capacity is used as the migration-out storage node, and the storage node with the lowest used capacity is used as the destination storage node;
[0197] A data block is extracted from the storage node to be migrated as the data block to be migrated.
[0198] In a possible implementation, the migration operation includes capacity expansion migration when a new storage node is added in the distributed cluster, and the above-mentioned determination module may also be used for:
[0199] For a storage node in the distributed cluster where a newly added disk exists, obtaining the used capacity of each disk under the storage node;
[0200] Whether to perform capacity expansion migration between disks is determined based on the number of disks under the storage node, the used capacity of each disk, and a preset disk capacity difference threshold. If it is determined that capacity expansion migration between disks is to be performed, the migration operation is triggered.
[0201] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.
[0202] In summary, the data migration storage method, device and management node provided by the embodiment of the present application obtain the data blocks to be migrated when the migration operation is triggered, and migrate the data blocks to be migrated to the target storage node and the target disk according to the data blocks in the target storage node and the target disk of the data blocks to be migrated when it is determined that the data blocks to be migrated meet the preset reliability requirements. When the preset reliability requirements are not met, it is detected whether the data blocks to be migrated form a ring structure. If so, the data blocks to be migrated are directly marked as the migration completion state. If not, wait for the next migration operation to be triggered. In this scheme, it is determined whether the data blocks to be migrated meet the preset reliability requirements based on the data blocks in the target storage node and the target disk, thereby avoiding the problem of subsequent anomalies that may be caused by the migration and thus causing data loss. In addition, for the data blocks to be migrated that form a special ring structure, the method of directly marking the migration success is adopted, which can avoid unnecessary migration work and ensure that each data block can be reasonably migrated.
[0203] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A data migration and storage method, characterized in that: Applied to a management node in a distributed cluster, the distributed cluster further comprising a plurality of storage nodes connected to the management node, each of the storage nodes comprising a plurality of disks, each of the disks being used to store data, each of the data being divided into a plurality of data blocks, the method comprising: When the migration operation is triggered, the data blocks to be migrated are obtained; According to the data blocks in the destination storage node and the destination disk of the data blocks to be migrated, detecting whether the data blocks to be migrated meet the preset reliability requirements, and if the preset reliability requirements are met, migrating the data blocks to be migrated to the destination storage node and the destination disk; If the preset reliability requirement is not met, detecting whether the data blocks to be migrated form a ring structure, wherein the ring structure is a structure in which the current storage nodes of two data blocks to be migrated are the destination storage nodes of each other; If a ring structure is formed, the two data blocks to be migrated are directly marked as being in a migration completion state; If a ring structure is not formed, wait for the next migration operation to be triggered; The migration operation includes a recovery migration operation after the storage node or disk is offline and then online. The step of detecting whether the data block to be migrated meets the preset reliability requirement according to the destination storage node and the data block in the destination disk of the data block to be migrated includes: Detect whether there is a data block with the same data as the data block to be migrated currently in a waiting for migration state; if not, detect whether there is a data block with the same data as the data block to be migrated in the destination disk; if not, then detect whether the number of data blocks with the same data as the data block to be migrated in the destination storage node is less than a preset maximum allowed number; if so, determine that the data block to be migrated meets the preset reliability requirements.
2. The data migration and storage method according to claim 1, characterized in that: The migration operation includes a recovery migration operation after the storage node or disk is offline and then online. The method also includes a step of recovering the data block when the storage node or disk is offline, which step includes: When it is determined that the storage node or disk is offline, the data information of the data blocks on the offline storage node or disk is obtained and stored in the recovery table; Scan the data information in the recovery table, and detect whether there are other data blocks belonging to the same data as the data block corresponding to the scanned data information in the recovery state; If it exists, skip the data information and perform scanning of the data information of the next data block; If it does not exist, the data block corresponding to the data information is sent to a destination storage node or a destination disk determined from other online storage nodes for recovery.
3. The data migration and storage method according to claim 2, characterized in that: The step of determining that the storage node or disk is offline includes: When it is detected that a storage node or a disk is offline and the offline time exceeds a preset time, it is determined that the storage node or the disk is offline.
4. The data migration and storage method according to claim 1, characterized in that: The migration operation includes capacity expansion migration when a new storage node is added in the distributed cluster. Before the step of triggering the migration operation, the method further includes: Obtaining the used capacity of each storage node currently existing in the distributed cluster; Based on the number of currently existing storage nodes, the used capacity of each currently existing storage node and the preset storage node capacity difference threshold, it is determined whether to perform capacity expansion migration between storage nodes. If it is determined to perform capacity expansion migration between storage nodes, the migration operation is triggered.
5. The data migration and storage method according to claim 4, characterized in that: The step of determining whether to perform capacity expansion migration between storage nodes according to the number of currently existing storage nodes, the currently used capacity of each storage node, and a preset capacity difference threshold comprises: The average used capacity of the storage nodes is calculated based on the total used capacity of all storage nodes in the distributed cluster and the number of currently existing storage nodes; For any storage node, based on the difference between the used capacity of the storage node and the average used capacity of the storage node, it is determined whether to perform capacity expansion migration between storage nodes.
6. The data migration and storage method according to claim 4, characterized in that: The step of obtaining the data block to be migrated includes: Comparing the used capacity of each storage node currently existing in the distributed cluster; The storage node with the highest used capacity is used as the migration-out storage node, and the storage node with the lowest used capacity is used as the destination storage node; A data block is extracted from the storage node to be migrated as the data block to be migrated.
7. The data migration and storage method according to claim 1, characterized in that: The migration operation includes capacity expansion migration when a new storage node is added in the distributed cluster. Before the step of triggering the migration operation, the method further includes: For a storage node in the distributed cluster where a newly added disk exists, obtaining the used capacity of each disk under the storage node; Whether to perform capacity expansion migration between disks is determined based on the number of disks under the storage node, the used capacity of each disk, and a preset disk capacity difference threshold. If it is determined that capacity expansion migration between disks is to be performed, the migration operation is triggered.
8. A data migration storage device, characterized in that: The method for data migration and storage according to any one of claims 1 to 7 is applied to a management node in a distributed cluster, wherein the distributed cluster further comprises a plurality of storage nodes connected to the management node, each of the storage nodes comprises a plurality of disks, each of the disks is used to store data, each of the data is divided into a plurality of data blocks, and the device comprises: The acquisition module is used to acquire the data blocks to be migrated when the migration operation is triggered; A first detection module, configured to detect whether the data block to be migrated meets a preset reliability requirement according to the destination storage node of the data block to be migrated and the data block in the destination disk; A migration module, configured to migrate the data block to be migrated to the destination storage node and the destination disk when the preset reliability requirement is met; A second detection module is used to detect whether the data blocks to be migrated form a ring structure when the preset reliability requirement is not met, wherein the ring structure is a structure in which the current storage nodes of two data blocks to be migrated are the destination storage nodes of each other; A marking module, used for directly marking the two data blocks to be migrated as being in a migration completion state when forming a ring structure; The waiting module is used to wait for the triggering of the next migration operation when a ring structure is not formed.
9. A management node, characterized in that: It includes one or more storage media and one or more processors communicating with the storage media, the one or more storage media store machine executable instructions executable by the processor, and when the management node runs, the processor executes the machine executable instructions to perform the method steps described in any one of claims 1-7.
