Method, device, equipment, medium and product for data migration

Abstract

The present disclosure provides a method, apparatus, device, medium and product for data migration. The method includes obtaining a first data block to be migrated and attribute information corresponding to the first data block to be migrated, wherein when the first data block to be migrated is stored in a first storage block, the attribute information is stored in a second storage block, and the first storage block corresponds to the second storage block; migrating the first data block to be migrated and the attribute information to a third storage block and a fourth storage block respectively; and updating the metadata information of the first data block to be migrated based on the third storage block and the attribute information. The method provided by the present disclosure can perform data migration and update the corresponding metadata information without traversing the KV data blocks. This will greatly reduce the consumption of system resources and improve the performance of the storage system.

Description

Technical Field

[0001] The present disclosure relates to the field of data migration, and more specifically, to a method, apparatus, device, medium and product for data migration. Background Art

[0002] In a distributed storage system, there will be operations to migrate data blocks from the source storage end to the target storage end, such as distributed storage systems that use append writes as the basic implementation method. For the write operation of the same data object (OBJ), from the business end perspective, the data is executed by overwriting, but from the storage end perspective, the data is executed by appending. This will cause multiple versions of data blocks for the same data object on the storage end, so for the same data object, only the data block with the largest version number (SEQ) needs to be retained, and the data blocks corresponding to other version numbers are garbage data.

[0003] When data block garbage collection is performed to reclaim system storage space, it is generally performed on the basis of a storage system unit (PLOG). Because different versions of data blocks under different data objects are stored under the same storage system unit (PLOG), data blocks with smaller version numbers can be directly released as garbage data, but data blocks with larger version numbers (that is, currently valid data of the corresponding data object) need to be migrated to the new target storage system unit. Since the source storage system unit and the target storage system unit ID are different, after the data block is migrated, in order to ensure the consistency of data block access, it is necessary to modify the storage system information stored in the data block metadata of the data object, and modify the storage unit address of the source data block to the data block address of the migrated target storage unit, so as to ensure that after space recovery is performed, the correct data information can still be accessed through the data block metadata of the data object.

[0004] Therefore, a new approach to data migration is needed. Summary of the invention

[0005] In view of the above problems, the present disclosure provides a new method for data migration. The method provided by the present disclosure can update the metadata set information of the data block to be migrated without traversing the database storing metadata information, so that the method provided by the present disclosure reduces the consumption of system resources and improves the storage system performance compared with the prior art.

[0006] An embodiment of the present disclosure provides a method for data migration, including: obtaining a first data block to be migrated and attribute information corresponding to the first data block to be migrated, wherein when the first data block to be migrated is stored in a first storage block, the attribute information is stored in a second storage block, and the first storage block corresponds to the second storage block; migrating the first data block to be migrated and the attribute information to a third storage block and a fourth storage block respectively; and updating metadata information of the first data block to be migrated based on the third storage block and the attribute information.

[0007] According to an embodiment of the present disclosure, the attribute information includes identification information of a data object corresponding to the first data block to be migrated and identification information of the first data block to be migrated.

[0008] According to an embodiment of the present disclosure, the updating of the metadata information of the first data block to be migrated based on the third storage block and the attribute information includes: updating the storage location information indicating the first data block to be migrated in the metadata information based on the third storage block; and obtaining the updated metadata information based on the attribute information and the storage location information.

[0009] According to an embodiment of the present disclosure, the method further includes: storing the first data block to be migrated and the attribute information in the first data block and the second data block respectively in a parallel manner.

[0010] According to an embodiment of the present disclosure, the method also includes: when the memory space occupied by the total attribute information corresponding to multiple data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size, each attribute information in the total attribute information including the attribute information is stored in a corresponding storage block respectively.

[0011] According to an embodiment of the present disclosure, the method also includes: storing the attribute information in a fifth storage block, loading the attribute information from the fifth storage block into a memory space; and determining whether the memory space occupied by the total attribute information corresponding to multiple data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size.

[0012] An embodiment of the present disclosure provides a device for data migration, including: an acquisition unit, which is configured to acquire a first data block to be migrated and attribute information corresponding to the first data block to be migrated, wherein when the first data block to be migrated is stored in a first storage block, the attribute information is stored in a second storage block, and the first storage block corresponds to the second storage block; a migration unit, which is configured to migrate the first data block to be migrated and the attribute information to a third storage block and a fourth storage block respectively; and an update unit, which is configured to update metadata information of the first data block to be migrated based on the third storage block and the attribute information.

[0013] According to an embodiment of the present disclosure, the attribute information includes identification information of a data object corresponding to the first data block to be migrated and identification information of the first data block to be migrated.

[0014] According to an embodiment of the present disclosure, the update unit includes: a first update sub-unit, which is configured to update the storage location information indicating the first data block to be migrated in the metadata information based on the third storage block; and a second update sub-unit, which is configured to obtain the updated metadata information based on the attribute information and the storage location information.

[0015] According to an embodiment of the present disclosure, the device further includes: a first storage unit configured to: store the first data block to be migrated and the attribute information in the first data block and the second data block respectively in a parallel manner.

[0016] According to an embodiment of the present disclosure, the device also includes: a second storage unit, which is configured to: when the memory space occupied by the total attribute information corresponding to multiple data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size, store each attribute information in the total attribute information including the attribute information in a corresponding storage block respectively.

[0017] According to an embodiment of the present disclosure, the device also includes: a third storage unit, which is configured to store the attribute information in a fifth storage block; a loading unit, which is configured to load the attribute information from the fifth storage block into the memory space; and a judgment unit, which is configured to judge whether the memory space occupied by the total attribute information corresponding to multiple data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size.

[0018] An embodiment of the present disclosure provides an apparatus for data migration, comprising: a component for obtaining a first data block to be migrated and attribute information corresponding to the first data block to be migrated, wherein when the first data block to be migrated is stored in a first storage block, the attribute information is stored in a second storage block, and the first storage block corresponds to the second storage block; a component for migrating the first data block to be migrated and the attribute information to a third storage block and a fourth storage block respectively; and a component for updating metadata information of the first data block to be migrated based on the third storage block and the attribute information.

[0019] An embodiment of the present disclosure provides a device for data migration, including: a processor, and a memory, wherein the memory stores computer executable instructions, and when the computer executable instructions are executed by the processor, the processor is prompted to execute any of the above methods.

[0020] An embodiment of the present disclosure provides a computer-readable recording medium storing computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor is prompted to execute any of the above methods.

[0021] An embodiment of the present disclosure provides a computer program product storing computer executable instructions, wherein the computer executable instructions, when executed by a processor, prompt the processor to perform any of the above methods.

[0022] The present disclosure provides a method, apparatus, device, medium and product for data migration. The method provided by the present disclosure can perform data migration and update the corresponding metadata information without traversing the KV data blocks. This will greatly reduce the consumption of system resources and improve the performance of the storage system. In addition, the method provided by the present disclosure can also perform data migration in such a way that the data is first merged and then written in batches, so that the method provided by the present disclosure not only meets the needs of batch refresh, but also can reduce the occupation of memory space, which further improves the performance of the storage system. In addition, the method provided by the present disclosure can first store the attribute information in the storage block and then write it into the memory space for batch processing, so that the method provided by the present disclosure avoids the data loss of the attribute information caused by abnormal conditions such as abnormal process exit or system power failure caused by directly putting the attribute information into the memory space and then writing it in batches. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some exemplary embodiments of the present disclosure, and a person of ordinary skill in the art can obtain other drawings based on these drawings without creative work.

[0024] Figure 1 A schematic diagram showing a data storage process according to an embodiment of the present disclosure is shown;

[0025] Figure 2 A schematic diagram of the data block migration and recovery process is shown, taking the recovery of PLOG.1 storage space as an example;

[0026] Figure 3 A flowchart of a method 300 for data migration according to an embodiment of the present disclosure is shown;

[0027] Figure 4A and Figure 4B A schematic diagram showing when attribute information corresponding to a data block is stored according to an embodiment of the present disclosure;

[0028] Figure 5 A schematic diagram showing a data storage process according to an embodiment of the present disclosure is shown;

[0029] Figure 6 A schematic diagram of data block migration according to an embodiment of the present disclosure is shown;

[0030] Figure 7 A flowchart of asynchronously writing data and synchronously writing attribute information according to an embodiment of the present disclosure is shown;

[0031] Figure 8 A schematic diagram showing the correspondence between a 64K DIF memory space and a PLOG according to an embodiment of the present disclosure is shown;

[0032] Fig. 9 A schematic diagram of a WAL type PLOG according to an embodiment of the present disclosure is shown;

[0033] Fig.10 According to an embodiment of the present disclosure, Figure 7 Add the above persistence operation to the flowchart;

[0034] Fig.11 A schematic diagram showing DIF recovery according to an embodiment of the present disclosure

[0035] Fig.12 A block diagram of an apparatus 1200 for data migration according to an embodiment of the present disclosure is shown;

[0036] Fig.13is a block diagram showing a device 1300 for data migration according to an embodiment of the present disclosure;

[0037] Fig.14 A schematic diagram showing a recording medium according to the present disclosure is shown. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0039] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0040] In the prior art, in a distributed storage system that uses append writing as the basic implementation method, in order to ensure that after executing the storage space recovery process, the correct data can continue to be accessed through the data block meta information (i.e. Key-Value, referred to as KV) of the data object, it is necessary to update the data block (BLOCK) address of the storage system unit (PLOG) recorded in the meta information KV. Specifically, when updating, it is necessary to traverse part or all of the KV records in the entire system, find the KV record item whose Value value completely matches the identification ID and offset address (i.e., PLOG_ID+OFFSET address) of the PLOG, and then keep the Key data unchanged, use the new PLOG_ID+OFFSET address to generate the Value data and save it. Thereby, the migration of the data block is realized. It can be seen from the migration method in the above-mentioned prior art that for a data block of a data object, when updating the corresponding KV record, it is necessary to traverse part or all of the KVs in the entire system, which will bring a lot of system overhead, resulting in problems such as a decrease in the processing capacity of the storage system.

[0041] In order to solve the above problems, the present disclosure provides a method for data migration. The method provided by the present disclosure can record the attribute information of the data block to be migrated (such as the identification information of the data object corresponding to the data block to be migrated and the identification information of the data block to be migrated) in the storage block in advance, so that when migrating the data block, the data block metadata (KV) of the data object can be reversely updated. Compared with the prior art, the method provided by the present disclosure reduces the consumption of system resources and improves the performance of the storage system.

[0042] Before describing the method provided by the present disclosure in detail with reference to the accompanying drawings, the following will introduce related technologies to facilitate a better understanding of the above method provided by the present disclosure.

[0043] According to an embodiment of the present disclosure, a file system (such as a distributed storage system) generally includes data and metadata.

[0044] As an example, metadata can be data about data, for example, metadata of a data block (META) can be data about a data block. Metadata can be information that mainly describes data attributes to support functions such as indicating storage location, historical data, resource search, file records, etc.

[0045] For example, metadata is usually managed through key-value pairs (KV), which are usually stored in a dedicated database (hereinafter referred to as KV database) for storage management.

[0046] A data object is any suitable data object in a file system. When storing data objects in a file system, the data object (OBJ) is usually divided into a certain size (e.g., 4M), and then when actually stored in a storage block, the data of this size is divided into multiple data blocks (BLOCK), and the size of the data block depends on the actual file system design, such as the size of the data block can be 8K. In this way, for data of this size, there will be a certain number of data blocks, such as the number equal to 4M / 8K (i.e., 512). The metadata information for one of the data blocks of the data object can be as follows:

[0047] Key = OBJ_ID+BLOCK_ID+SEQ;

[0048] Value = PLOG_ID+OFFSET.

[0049] Among them, OBJ_ID indicates the identifier of the data object. BLOCK_ID indicates the identifier of the data block. SEQ indicates the version number, where SEQ increases by 1 each time a KV record is created, and when there are two Key values ​​with exactly the same information except SEQ, the KV with a larger SEQ is a valid record. PLOG_ID indicates the identifier of the storage system unit, and OFFSET indicates the corresponding offset.

[0050] Different storage systems have different designs for the storage system unit (PLOG), but they usually include the following two parameters:

[0051] Storage unit size: such as 128M.

[0052] The block size in the storage unit: such as 8K.

[0053] Figure 1 A schematic diagram of a data storage process according to an embodiment of the present disclosure is shown.

[0054] Reference Figure 1 , first write the first data block BLOCK.1 in the first data object OBJ.1, as follows:

[0055] SEQ.2: Write OBJ.1 -> BLOCK.1

[0056] The first 8K data block of OBJ.1 is stored in the 8K BLOCK with an offset of 1 (i.e., OFFSET.1) in the first storage system unit (PLOG.1). At this time, the metadata information for the first 8K data block of OBJ.1 is as follows:

[0057] Key = OBJ.1 + BLOCK.1 + SEQ.2

[0058] Value = PLOG.1 + OFFSET.1

[0059] Next, write the second data block BLOCK.2 in OBJ.1 as follows:

[0060] SEQ.6: Write OBJ.1 -> BLOCK.2

[0061] The second 8K data of OBJ.1 is stored in the 8K BLOCK at offset OFFSET.2 in PLOG.1. At this time, the metadata information for the second 8K data block of OBJ.1 is as follows:

[0062] Key = OBJ.1 + BLOCK.2 + SEQ.6

[0063] Value = PLOG.1 + OFFSET.2

[0064] Next, write the first data block BLOCK.1 in the second data object OBJ.2 as follows:

[0065] SEQ.7: Write OBJ.2 -> BLOCK.1

[0066] The first 8K data of OBJ.2 is stored in the 8K BLOCK at offset OFFSET.3 in PLOG.1. At this time, the metadata information for the first 8K data block of OBJ.2 is as follows:

[0067] Key = OBJ.2 + BLOCK.1 + SEQ.7

[0068] Value = PLOG.1 + OFFSET.3

[0069] Next, write the second data block BLOCK.2 in OBJ.2 as follows:

[0070] SEQ.8: Write OBJ.2 -> BLOCK.2

[0071] The second 8K data of OBJ.2 is stored in the 8K BLOCK at offset OFFSET.4 in PLOG.1. At this time, the metadata information for the second 8K data block of OBJ.2 is as follows:

[0072] Key = OBJ.2 + BLOCK.2 + SEQ.8

[0073] Value = PLOG.1 + OFFSET.4

[0074] Next, write the Nth data block BLOCK.N in OBJ.2 as follows:

[0075] SEQ.10: Write OBJ.2 -> BLOCK.N

[0076] The Nth 8K data of OBJ.2 is stored in the 8K BLOCK with offset OFFSET.N in PLOG.1. At this time, the metadata information for the Nth 8K data block of OBJ.2 is as follows:

[0077] Key = OBJ.2 + BLOCK.N + SEQ.10

[0078] Value = PLOG.1 + OFFSET.N

[0079] Then, write the first data block BLOCK.1 in OBJ.1 again, as follows:

[0080] SEQ.13: Rewrite OBJ.1 -> BLOCK.1

[0081] The first 8K data of OBJ.1 is stored in the 8K BLOCK with offset OFFSET.2 in PLOG.2. At this time, the metadata information for the first 8K data block of OBJ.1 is as follows:

[0082] Key = OBJ.1 + BLOCK.1 + SEQ.13

[0083] Value = PLOG.2 + OFFSET.2

[0084] At this time, the data block in the storage system unit corresponding to SEQ.2 is garbage data.

[0085] Then, write the first data block BLOCK.1 in OBJ.2 again, as follows:

[0086] SEQ.28: Rewrite OBJ.2 -> BLOCK.1

[0087] The first 8K data of OBJ.2 is stored in the 8K BLOCK at offset OFFSET.3 in PLOG.2. At this time, the metadata information for the first 8K data block of OBJ.2 is as follows:

[0088] Key = OBJ.2 + BLOCK.1 + SEQ.28

[0089] Value = PLOG.2 + OFFSET.3

[0090] At this time, the data block in the storage system unit corresponding to SEQ.7 is garbage data.

[0091] Then, write the second data block BLOCK.2 in OBJ.1 again, as follows:

[0092] SEQ.31: Rewrite OBJ.1 -> BLOCK.2

[0093] The second 8K data of OBJ.1 is stored in the 8K BLOCK at offset OFFSET.4 in PLOG.2. At this time, the metadata information for the second 8K data block of OBJ.1 is as follows:

[0094] Key = OBJ.1 + BLOCK.2 + SEQ.31

[0095] Value = PLOG.2 + OFFSET.4

[0096] At this time, the data block in the storage system unit corresponding to SEQ.6 is garbage data.

[0097] As introduced in the background technology, when data block garbage collection is performed to reclaim system storage space, it is generally performed based on a PLOG. Because different versions of data blocks under different data objects are stored under the same PLOG, data blocks with smaller version numbers can be directly released as garbage data, but data blocks with larger version numbers (i.e., currently valid data of the corresponding data object) need to be migrated to a new target storage system unit.

[0098] Specifically, refer to Figure 2 , Figure 2 A schematic diagram of the data block migration and recovery process is shown, taking the recovery of PLOG.1 storage space as an example.

[0099] First, the storage system creates a new PLOG, for example, PLOG.3.

[0100] Next, the storage system will mark the garbage data blocks in PLOG.1: OFFSET.1 OFFSET.2 OFFSET.3.

[0101] Next, the storage system copies the 8K BLOCK data at OFFSET.4 and OFFSET.N in PLOG.1 to OFFSET.1 and OFFSET.2 in PLOG.3, that is, migrate 8K data.

[0102] Then, traverse the KV database, find the Key with Value=PLOG.1+OFFSET.4, extract the OBJ and BLOCK information from the Key, that is, OBJ.2 and BLOCK.2; then regenerate the KV record, Key=OBJ.2+BLOCK.2+SEQ.36, Value=PLOG.3+OFFSET.1, that is, lookup DB to get KV to create new KV, and finally write the newly generated KV to the KV database, so that the migrated 8K BLOCK data can be correctly accessed through the newly generated KV record. Similarly, according to the above operation method, perform data migration for PLOG.1+OFFSET.N and generate a new KV record.

[0103] From the above migration method, we can see that in order to migrate an 8K BLOCK data, we need to traverse the KV database, find the corresponding Key-Value, parse the required fields from the Key, and then regenerate a new KV. Traversing the KV database will consume a lot of system resources, resulting in a decrease in storage system performance.

[0104] In order to solve the above problems, the present disclosure provides a novel method for data migration as described above. Next, the method provided by the present disclosure will be described in detail with reference to the accompanying drawings.

[0105] Figure 3 A flow chart of a method 300 for data migration according to an embodiment of the present disclosure is shown. The method 300 provided by the present disclosure can be executed by any suitable device in the storage system, such as a processor, a processing unit, etc. Alternatively, the method 300 provided by the present disclosure can be executed by an external device interconnected with the storage system, such as an external processor, an external processing unit, etc.

[0106] Reference Figure 3 In step S310, a first data block to be migrated and attribute information corresponding to the first data block to be migrated may be obtained.

[0107] According to an embodiment of the present disclosure, when the first data block to be migrated is stored in a first storage block, the attribute information is stored in a second storage block, and the first storage block corresponds to the second storage block.

[0108] According to an embodiment of the present disclosure, the attribute information may include identification information of a data object corresponding to the first data block to be migrated and identification information of the first data block to be migrated.

[0109] As an example, the first data block to be migrated may be a combination of the above Figure 1 and 2Any data blocks to be migrated in PLOG.1 and PLOG.2 described, such as the 8K data blocks BLOCK of OFFSET.4 and OFFSET.N in PLOG.1.

[0110] As an example, the attribute information may include identification information of a data object corresponding to the first data block to be migrated (eg, OBJ.1 or OBJ.2) and identification information of the first data block to be migrated (eg, BLOCK.1, BLOCK.2, BLOCK.3, . . . BLOCK.N).

[0111] As an example, the first data block to be migrated may be stored in a first storage block (i.e., PLOG.1+OFFSET.1) in PLOG.1 with an offset of 1. At this time, the attribute information may be stored in a second storage block corresponding to the first storage block, wherein the second storage block may be a storage block at any offset in any PLOG.

[0112] As an example, the first data block to be migrated and the attribute information may be stored in the first data block and the second data block respectively in a parallel manner, so as to reduce the time consumption caused by the write operation.

[0113] Continue to refer to Figure 3 In step S320, the first data block to be migrated and the attribute information may be migrated to a third storage block and a fourth storage block respectively.

[0114] As an example, the first data block to be migrated stored in the first storage block may be migrated to the third storage block, and the attribute information stored in the second storage block may be migrated to the fourth storage block in any suitable manner.

[0115] In step S330, metadata information of the first data block to be migrated may be updated based on the third storage block and the attribute information.

[0116] According to an embodiment of the present disclosure, updating the metadata information of the first data block to be migrated based on the third storage block and the attribute information includes: updating the storage location information indicating the first data block to be migrated in the metadata information based on the third storage block; and obtaining the updated metadata information based on the attribute information and the storage location information.

[0117] As an example, the third storage block may be a combination of the above Figure 2The offset of PLOG.3 described is 1 for an 8K storage block. At this time, the metadata information of the first data block to be migrated indicates that the storage location information of the first data block to be migrated should include PLOG.3 + OFFSET.1. In this case, the K value in the metadata information KV can be updated to include PLOG.3 + OFFSET.1. The attribute information at this time may include, for example: OBJ.2 + BLOCK2. Based on the above attribute information and storage location information, the updated metadata information can be obtained. As follows, the updated metadata information at this time can be:

[0118] Key = OBJ.2 + BLOCK.2 + SEQ.36

[0119] Value = PLOG.3 + OFFSET.1

[0120] With the updated metadata information, you can prepare to find the migrated data content.

[0121] From the above content, it can be seen that the method provided by the present disclosure can perform data migration and update corresponding metadata information without traversing KV data blocks, which will greatly reduce the consumption of system resources and improve the performance of the storage system.

[0122] According to an embodiment of the present disclosure, the method 300 may also include storing each attribute information in the total attribute information including the attribute information to a corresponding storage block when the memory space occupied by the total attribute information corresponding to multiple data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size.

[0123] As an example, the predetermined threshold size can be any suitable size set based on actual needs, for example, it can be 64K. Assuming that an attribute information occupies 64B, 1024 attribute information (i.e., 64K / 64B) can be stored within the predetermined size 64K. In the above case, only when the predetermined threshold size 64K is reached, all attribute information including the above attribute information will be stored in the corresponding storage blocks respectively, instead of immediately storing the above attribute information in the corresponding storage block. In this way of merging data first and then writing in batches, there is no need to frequently create and destroy memory space (such as cache) for an attribute information, which not only meets the needs of batch refresh, but also reduces the occupancy of memory space. Thereby further improving the performance of the storage system.

[0124] According to an embodiment of the present disclosure, the method 300 may also include storing the attribute information in a fifth storage block, loading the attribute information from the fifth storage block into a memory space; judging whether the memory space occupied by the total attribute information corresponding to the multiple data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size. That is, the method provided by the present disclosure may also store the attribute information in a storage block first, and then read the attribute information from the storage block into the content space to perform the above batch writing operation, that is, first judging whether the memory space occupied by the total attribute information corresponding to the multiple data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size; if the predetermined threshold size is reached, performing the above-mentioned storage of each attribute information in the total attribute information including the attribute information to the corresponding storage block; if the predetermined threshold size is not reached, continue to wait. In the above manner, the data loss of the attribute information caused by abnormal conditions such as abnormal process exit or system power failure caused by directly putting the attribute information into the memory space and then writing it in batches is avoided.

[0125] The above combination Figures 1 to 3 The method provided by the present disclosure is described in detail. The method provided by the present disclosure can perform data migration and update the corresponding metadata information without traversing the KV data block. This will greatly reduce the consumption of system resources and improve the performance of the storage system. In addition, the method provided by the present disclosure can also perform data migration in such a way that the data is first merged and then written in batches, so that the method provided by the present disclosure not only meets the needs of batch refresh, but also can reduce the occupation of memory space, which in turn further improves the performance of the storage system. In addition, the method provided by the present disclosure can first store the attribute information in the storage block and then write it into the memory space for batch processing, so that the method provided by the present disclosure avoids the loss of attribute information data caused by abnormal situations such as abnormal process exit or system power failure caused by directly putting the attribute information into the memory space and then writing it in batches.

[0126] In order to facilitate a better understanding of the above method provided by the present disclosure, the above method provided by the present disclosure will be further explained in the form of examples.

[0127] Figure 4A and Figure 4B A schematic diagram is shown when attribute information corresponding to a data block is stored according to an embodiment of the present disclosure.

[0128] Reference Figure 4A After the attribute information corresponding to the data block is introduced and stored together in the storage block, the attribute information at this time can indirectly indicate the corresponding relationship between the data block meta information and the data block in the storage system unit. Figure 4AIn this way, each time when a data object, such as an 8K data block, is written, an additional DIF record area will be added for the corresponding 8K BLOCK data in the current PLOG, which is used to mark the attribute information corresponding to the 8K BLOCK data of the current PLOG, that is, the identification information of the data object OBJ corresponding to the data block and the identification information of the data block BLOCK.

[0129] The DIF record area can be located in the storage space of the current PLOG ID+1, or in the storage space of the current PLOG ID-1. Each DIF field (i.e., the storage area that records each attribute information) can occupy 64 bytes. In this way, if the PLOG size is 128MB, and one BLOCK is 8K, then there are 16384 BLOCKs in total, and 16384 DIF fields are required at this time. These DIF fields require a total storage area of ​​16384*64Byte=1MB. That is, a 128MB PLOG needs to occupy an additional 1MB of DIF storage space. For subsequent expansion, the current DIF can be set to occupy 2MB of space.

[0130] like Figure 4B As shown, the OFFSET of the total DIF storage space including the above-mentioned various DIF fields can correspond one-to-one with the OFFSET in the PLOG, that is, for example, the 64 bytes of OFFSET.1 of the DIF corresponds to the BLOCK of OFFSET.1 of the PLOG.

[0131] The data storage process at this time can be as follows Figure 5 shown. Figure 5 A schematic diagram of a data storage process according to an embodiment of the present disclosure is shown.

[0132] Reference Figure 5 At 501, an 8K data block BLOCK of a data object OBJ is written based on the user's operation, namely, WRITE OBJ 8K Block. Then a write instruction WRITE BOJ.N+BLOCK.N is sent to the Write OBJ process.

[0133] At 502, a K value in the metadata information KV of the data block is created, that is, create Key = BOJ.N+BLOCK.N+SEQ.N.

[0134] At 503, information including the identifier of the currently available PLOG and the corresponding offset is obtained, namely, getPlogID and OFFSET.

[0135] In 504, after the obtained information is transmitted to the PLOG process, the PLOG process selects the corresponding PLOG and the current offset, that is, select PLOG and current OFFSRT.

[0136] At 505 , after receiving the selected PLOG.N and OFFSET.N, the Write OBJ process creates the V value in KV, ie, create Value=PLOG.N+OFFSET.N.

[0137] In 506, the 8K data block BLOCK of the data object OBJ is written to the storage block corresponding to the OFFSET of the selected PLOG, that is, Write 8K data to PLOG.

[0138] In 507, after receiving the above write instruction, the PLOG process prepares to write an 8K data block, that is, Write 8K BLOCK.

[0139] At 508, the PLOG process writes the 8K data block of the data object OBJ to the storage block corresponding to the OFFSET of the selected PLOG, that is, Write Data to the OFFSET of PLOG.

[0140] If writing fails, the corresponding error information will be fed back to the user, that is, the failure will be fed back to the write object (Write OBJ) process first, that is, Write PLOG ERROR, and then the error will be fed back to the user, that is, return ERROR.

[0141] If the writing is successful, then at 509 , after receiving the writing success indication (ie, return ok), the Write OBJ process starts writing the attribute information, ie, Write DIF.

[0142] At 510, after receiving PLOG.N+OBJ.N+BLOCK.N, the attribute information process (PLOG.DIF) writes OBJ.N+BLOCK.N to the same position as the OFFSET of PLOG in the DIF storage space, that is, Write DIF data.

[0143] If writing fails, the corresponding error information will be fed back to the user, that is, the failure will be fed back to the write object (Write OBJ) process first, that is, Write DIF ERROR, and then the error will be fed back to the user, that is, return ERROR.

[0144] If the write is successful, then in 511, after receiving the write success indication (i.e., return ok), the write object (Write OBJ) process writes the corresponding metadata information KV in the KV database DB, i.e., Write KV to DB. If the write to DB fails, an error message will be fed back to the user, i.e., Write DB ERROR. If the write to DB is successful, a success indication will be fed back to the user, i.e., return OK.

[0145] It should be noted that the Write OBJ process, PLOG process and attribute information process (PLOG.DIF) introduced above and to be introduced next can be the same process or different processes.

[0146] Figure 6 A schematic diagram of data block migration according to an embodiment of the present disclosure is shown.

[0147] Reference Figure 6 , the storage system will create a new PLOG, for example, PLOG.3.

[0148] Next, the storage system will mark the garbage data blocks in PLOG.1: OFFSET.1 OFFSET.2 OFFSET.3.

[0149] Next, the storage system copies the 8K BLOCK data at OFFSET.4 and OFFSET.N in PLOG.1 to OFFSET.1 and OFFSET.2 in PLOG.3, that is, migrate 8K data.

[0150] Finally, the storage system will retrieve OBJ.2+BLOCK.2 according to the attribute information data (i.e., DIF data) corresponding to PLOG.1+OFFSET.4, and then directly regenerate the KV record, i.e., Key=OBJ.2+BLOCK.2+SEQ.36, Value=PLOG.3+OFFSET.1, i.e., the instruction use DIF to create new KV in the figure; finally, the newly generated KV is written into the KV database. Similarly, according to the above operation method, data migration is performed for PLOG.1+OFFSET.N and a new KV record is generated.

[0151] It can be seen from the above content that the method provided by the present disclosure is Figure 2 The method shown, after introducing the above-mentioned DIF mechanism, does not need to traverse the KV database, and can directly use the DIF data to generate new KV records, which will greatly reduce the consumption of system resources and improve the performance of the storage system.

[0152] After the DIF mechanism is introduced, garbage data recovery can be accelerated, but it will cause the original one Write operation to become two Write operations, that is, Write Data + Write DIF. This will inevitably increase the time consumption of the Write operation. In order to reduce the impact of Write DIF on system performance, the method provided by the present disclosure is further optimized, that is, the asynchronous Write Data + synchronous Write DIF method is adopted. Figure 7 As shown, Figure 7 A flowchart of asynchronously writing data and synchronously writing attribute information according to an embodiment of the present disclosure is shown.

[0153] Reference Figure 7 , Figure 7 701-706 in combination with the above Figure 5 The descriptions of 501-506 are similar and will not be repeated here.

[0154] In 707, after receiving the write data (i.e., Write 8K BLOCK), the PLOG process starts the background task task to execute the data write process; if the task task fails to start, an error is returned (i.e., return create task ERROR); if the task task is successfully started, it returns directly (i.e., return). In this way, after the background task task is executed, the execution result of the write data task task is returned through a callback (such as callback). In this way, Figure 5 The synchronous data writing operation shown in is changed to the asynchronous data writing operation.

[0155] After the front end receives the task of writing data and successfully starts, it executes the synchronous Write DIF process. That is, at 709, Write OBJ starts writing attribute information, that is, Write DIF. At 710, after the attribute information process (PLOG.DIF) receives PLOG.N+OBJ.N+BLOCK.N, it writes OBJ.N+BLOCK.N to the same position of OFFSET of PLOG in the DIF storage space, that is, Write DIF data.

[0156] At 711, after receiving the indication of successful writing of DIF data (i.e., return OK / ERROR), check whether the asynchronous writing data task task is completed; if not, continue to wait for the completion of the writing data task, and then check the execution results of the asynchronous writing data task task and the synchronous writing DIF; if an exception occurs, return an error (i.e., return ERROR). After the asynchronous writing data task task and the synchronous writing DIF are both executed successfully (i.e., ALL is OK), save the KV record to the database DB at 712, i.e., Write KV to DB. If writing to DB fails, an error message will be fed back to the user, i.e., Write DB ERROR. If writing to DB is successful, a success indication will be fed back to the user, i.e., return OK.

[0157] Combining the above Figure 7 It can be seen from the described process that by changing the two synchronous write operations to one asynchronous and one synchronous operation, that is, writing data and writing DIF are executed in parallel, the time consumption is equivalent to one write operation, thus offsetting the performance overhead brought by the introduction of the DIF mechanism.

[0158] In a storage system based on append-write, in order to speed up data writing performance, if conditions permit, the written data will be merged and then written in batches.

[0159] Generally, the size of a DIF data is 64 bytes. If a persistence operation is performed every time a DIF data is written (that is, stored in the corresponding storage block), the performance of a small-granularity data storage such as 64 bytes is much worse than that of a larger storage granularity (such as 4K, 8K). Therefore, in order to improve the persistence performance of DIF data, multiple DIF data will be merged and batch persistence refresh operations will be performed.

[0160] When merging multiple DIF data, the DIF data is usually temporarily cached in the memory space. For example, if a PLOG size is 128MB, the DIF occupies 2MB of space. At the same time, thousands or even tens of thousands of PLOGs will be opened in the storage system at the same time. If a 2MB DIF space is allocated in the memory for each PLOG, a large amount of system memory will be occupied, squeezing the memory space of other modules, and ultimately causing a decrease in storage performance.

[0161] In order to achieve batch refresh and solve the problem of DIF data memory space occupation, the present disclosure sets the DIF memory space cache corresponding to a PLOG to, for example, 64K. At this time, the DIF data that can be accommodated is 64K / 64Byte=1024. In this case, when the number of merged DIFs reaches 1024, the flush DIF process is started to persist the DIF data.

[0162] When flush DIF is completed, the 64K DIF cache data is cleared, and the 64K DIF cache start OFFSET is changed to 1025 and the end OFFSET is changed to 2048, that is, the DIF cache points to the PLOG.DIF represented by the next group of 64K DIF. This continues until the DIF cache points to the last group of 64K DIF in PLOG. Figure 8 With this operation flow, the 64K DIF cache only needs to be created once when the PLOG is written for the first time, and the DIF cache can be reused later without the need to frequently create and destroy the DIF cache. This not only meets the requirements of batch refresh, but also reduces the memory space occupied by the DIF cache.

[0163] Under normal writing process, DIF data is stored in memory space, and only when the number of DIFs reaches 1024, will the flush operation be performed to persist the DIF data in the cache. When DIF data is stored in memory space and persistence has not yet been performed, if abnormal conditions such as abnormal process exit or system power failure occur, the DIF data cached in the memory space will be lost due to the irrecoverability of memory data.

[0164] To this end, as mentioned above, Figure 3 As described above, the method provided by the present disclosure can first store the attribute information (ie, DIF data) in a storage block, and then read the attribute information from the storage block into the content space.

[0165] For example, the DIF cache data loss problem can be solved by first storing the DIF data in the storage block and then reading it into the content space through the write-ahead log (WAL) method.

[0166] Specifically, while writing DIF data to memory, DIF data is saved to WAL-type PLOG (i.e., DIF WAL PLOG) in an append-write manner. DIF WAL uses WAL-type PLOG as a persistent carrier. Based on a 128MB PLOG, a WAL-type PLOG can store 128MB / 64Byte= 2097152 DIF data, and a PLOG has 16384 DIFs, so a WAL PLOG can save DIF data of 128 PLOGs. To avoid frequent creation and destruction of WAL-type PLOGs (creation and destruction operations will increase system resource overhead), a DIF WAL will record DIF data under multiple PLOGs.

[0167] When the PLOG of DIF WAL is full, a new WAL PLOG will be created to continue persisting DIF data. The old DIF WALPLOG will be released after all DIF Data associated with this DIF WAL PLOG is flushed. The persistent DIF data of DIF WAL is only used for DIF recovery after an exception. The DIF cache flush process under the normal process still uses 64K as the benchmark to perform persistence of DIF data to the corresponding PLOG. After adopting the DIF WAL method, compared with the previous DIF writing method, since DIF WAL is a batch write operation after merging DIF data, it is equivalent to changing multiple small-grained Write 64Byte DIF Data to a single Write operation with a larger write granularity, so it can significantly improve the DIF write performance. Fig. 9 As shown, a DIFWAL.PLOG can store DIF data with corresponding OFFSET of the same or different PLOGs. One or more DIF data can be regarded as a DIF WAL record.

[0168] Fig.10 According to an embodiment of the present disclosure, Figure 7 The flowchart after adding the above persistence operation on this basis.

[0169] Reference Fig.10 , Fig.10 1001-1009 in combination with the above Figure 7 The descriptions of 701-709 are similar and will not be repeated here.

[0170] At 1010, after receiving a request to merge DIF data (Merge DIF Data), the persistence process (DIFWAL) writes the DIF data to the PLOG of the WAL type, that is, Write DIF Data to WAL. At 1011, the above persistent data is then loaded into the memory, that is, Write DIF Data to Cache.

[0171] At 1012 , it is determined whether the DIF cache size has reached a predetermined threshold size of 64K (ie, DIF Cache size is 64k)?

[0172] When it reaches 64K (ie, YES), at 1013, the 64K DIF data is batch-flushed, ie, Flush DIF Data, and a success indication is returned (ie, return OK / ERROR).

[0173] If 64K is not reached (ie, NO), continue to wait. When all data has been refreshed, feedback completion indication is given to the Write OBJ process, ie, return OK.

[0174] At 1014, the execution results of writing data and writing DIF WAL data are checked. If an exception occurs, an error is returned (i.e., return ERROR). After writing data and writing DIF WAL data are both successful (i.e., ALL is OK), the KV record is saved to the database DB at 1015, i.e., Write KV to DB. If writing to DB fails, an error message is fed back to the user, i.e., Write DB ERROR. If writing to DB succeeds, a success indication is fed back to the user, i.e., return OK.

[0175] When the above persistence mechanism is entered, when the process exits abnormally or the system loses power, when the program is started again, the DIF cache will be rebuilt by reading the DIF WAL PLOG information. In order to speed up the recovery process, the DIF WAL recovery operation will be executed as a background task.

[0176] like Fig.11 As shown, first, at 1101, the main program (Main) process creates a DIF WAL recovery background task, namely, Start DIF WAL Recover Background Task.

[0177] At 1102, in the asynchronous Async mode, the global table process (Global Table) obtains the DIFWAL PLOG address from the global list, namely, Get DIF WAL PLOGs Address.

[0178] At 1103 , the DIF WAL process reads the DIF WAL PLOG data (ie, Read DIF WAL PLOG Data), and parses the DIF Data at 1104 , ie, Parse DIF Data from WAL.

[0179] At 1105, the parsed DIF Data is written into the cache, namely, Write DIF Data to Cache.

[0180] After all DIF WAL data is recovered to the cache (ie, ALL DIF Data is Recovered) at 1106, the above-mentioned DIF Flush process is executed to perform a persistent DIF operation (ie, Write DIFdata) at 1107 in the PLOG.DIF process.

[0181] Finally, at 1108, a new DIF WAL object is created to provide the Write DIF WAL service under the normal process, namely, Create DIF WAL Service.

[0182] It should be noted that the above-mentioned main program (Main) process, global table process (Global Table), DIF WAL process and PLOG.DIF process can be the same process or different processes.

[0183] In addition to the above method, the present disclosure also provides a device for data migration. The device provided by the present disclosure will be described below in conjunction with the accompanying drawings. The above description of the method is also applicable to the device to be described below, unless otherwise explicitly stated.

[0184] Fig.12 A block diagram of an apparatus 1200 for data migration according to an embodiment of the present disclosure is shown.

[0185] Reference Fig.12 , the device 1200 may include an acquisition unit 1210 , a migration unit 1220 and an update unit 1230 .

[0186] According to an embodiment of the present disclosure, the acquiring unit 1210 may be configured to acquire the first data block to be migrated and attribute information corresponding to the first data block to be migrated.

[0187] According to an embodiment of the present disclosure, when the first data block to be migrated is stored in a first storage block, the attribute information is stored in a second storage block, and the first storage block corresponds to the second storage block.

[0188] According to an embodiment of the present disclosure, the attribute information may include identification information of a data object corresponding to the first data block to be migrated and identification information of the first data block to be migrated.

[0189] As an example, the first data block to be migrated may be a combination of the above Figure 1 and 2 Any data blocks to be migrated in PLOG.1 and PLOG.2 described, such as the 8K data blocks BLOCK of OFFSET.4 and OFFSET.N in PLOG.1.

[0190] As an example, the attribute information may include identification information of a data object corresponding to the first data block to be migrated (eg, OBJ.1 or OBJ.2) and identification information of the first data block to be migrated (eg, BLOCK.1, BLOCK.2, BLOCK.3, . . . BLOCK.N).

[0191] As an example, the first data block to be migrated may be stored in a first storage block (i.e., PLOG.1+OFFSET.1) in PLOG.1 with an offset of 1. At this time, the attribute information may be stored in a second storage block corresponding to the first storage block, wherein the second storage block may be a storage block at any offset in any PLOG.

[0192] According to an embodiment of the present disclosure, the apparatus 1200 may further include a first storage unit, which may be configured to store the first to-be-migrated data block and the attribute information in the first data block and the second data block in parallel, respectively, so as to reduce the time consumption caused by the write operation.

[0193] Continue to refer to Fig.12 The migration unit 1220 may be configured to migrate the first data block to be migrated and the attribute information to the third storage block and the fourth storage block respectively.

[0194] As an example, the first data block to be migrated stored in the first storage block may be migrated to the third storage block, and the attribute information stored in the second storage block may be migrated to the fourth storage block in any suitable manner.

[0195] According to an embodiment of the present disclosure, the updating unit 1230 may be configured to update metadata information of the first data block to be migrated based on the third storage block and the attribute information.

[0196] According to an embodiment of the present disclosure, the update unit 1230 may include: a first update subunit and a second update subunit, wherein the first update subunit may be configured to update the storage location information indicating the first data block to be migrated in the metadata information based on the third storage block; the second update subunit may be configured to obtain the updated metadata information based on the attribute information and the storage location information.

[0197] As an example, the third storage block may be a combination of the above Figure 2 The offset of PLOG.3 described is 1 for an 8K storage block. At this time, the metadata information of the first data block to be migrated indicates that the storage location information of the first data block to be migrated should include PLOG.3 + OFFSET.1. In this case, the K value in the metadata information KV can be updated to include PLOG.3 + OFFSET.1. The attribute information at this time may include, for example: OBJ.2 + BLOCK2. Based on the above attribute information and storage location information, the updated metadata information can be obtained. As follows, the updated metadata information at this time can be:

[0198] Key = OBJ.2 + BLOCK.2 + SEQ.36

[0199] Value = PLOG.3 + OFFSET.1

[0200] With the updated metadata information, you can prepare to find the migrated data content.

[0201] From the above content, it can be seen that the device provided by the present disclosure can perform data migration and update corresponding metadata information without traversing KV data blocks, which will greatly reduce the consumption of system resources and improve the performance of the storage system.

[0202] According to an embodiment of the present disclosure, the device 1200 may also include a second storage unit, which may be configured to store each attribute information in the total attribute information including the attribute information to a corresponding storage block when the memory space occupied by the total attribute information corresponding to multiple data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size.

[0203] As an example, the predetermined threshold size can be any suitable size set based on actual needs, for example, it can be 64K. Assuming that an attribute information occupies 64B, 1024 attribute information (i.e., 64K / 64B) can be stored within the predetermined size 64K. In the above case, only when the predetermined threshold size 64K is reached, all attribute information including the above attribute information will be stored in the corresponding storage blocks respectively, instead of immediately storing the above attribute information in the corresponding storage block. In this way of merging data first and then writing in batches, there is no need to frequently create and destroy memory space (such as cache) for an attribute information, which not only meets the needs of batch refresh, but also reduces the occupancy of memory space. Thereby further improving the performance of the storage system.

[0204] According to an embodiment of the present disclosure, the device 1200 may further include a third storage unit, a loading unit, and a judgment unit, wherein the third storage unit may be configured to store the attribute information in a fifth storage block; the loading unit may be configured to load the attribute information from the fifth storage block into the memory space; the judgment unit may be configured to judge whether the memory space occupied by the total attribute information corresponding to the multiple data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size. That is, the device provided by the present disclosure may also store the attribute information in a storage block first, and then read the attribute information from the storage block into the content space to perform the above batch writing operation, that is, first judge whether the memory space occupied by the total attribute information corresponding to the multiple data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size; if the predetermined threshold size is reached, perform the above-mentioned storage of each attribute information in the total attribute information including the attribute information to the corresponding storage block; if the predetermined threshold size is not reached, continue to wait. In the above manner, the data loss of the attribute information caused by abnormal conditions such as abnormal process exit or system power failure caused by directly putting the attribute information into the memory space and then writing it in batches is avoided.

[0205] It should be noted that, although the device 1200 is divided into units for respectively performing corresponding processes in the above description, it is clear to those skilled in the art that the processes performed by each unit may also be performed without any specific unit division or without clear demarcation between the units. Fig.12 The described apparatus is not limited to including the above-described units, but some other units (eg, data processing units, etc.) may be added as needed, or the above units may also be combined.

[0206] It can be seen from the device provided by the present disclosure described in detail above in combination with the accompanying drawings that the device provided by the present disclosure can perform data migration and update the corresponding metadata information without traversing the KV data blocks. This will greatly reduce the consumption of system resources and improve the performance of the storage system. In addition, the device provided by the present disclosure can also perform data migration in such a way that the data is first merged and then written in batches, so that the device provided by the present disclosure not only meets the needs of batch refresh, but also can reduce the memory space occupied, which further improves the storage system performance. In addition, the device provided by the present disclosure can first store the attribute information in the storage block and then write it to the memory space for batch processing, so that the device provided by the present disclosure avoids the loss of attribute information data caused by abnormal situations such as abnormal process exit or system power failure caused by directly putting the attribute information into the memory space and then writing it in batches.

[0207] In addition to the above-mentioned method and device, the present disclosure also provides another device. The above-mentioned description of the method and device is also applicable to another device to be described below, unless otherwise explicitly stated. A device for data migration, comprising: a component for obtaining a first data block to be migrated and attribute information corresponding to the first data block to be migrated, wherein when the first data block to be migrated is stored in a first storage block, the attribute information is stored in a second storage block, and the first storage block corresponds to the second storage block; a component for migrating the first data block to be migrated and the attribute information to a third storage block and a fourth storage block respectively; and a component for updating the metadata information of the first data block to be migrated based on the third storage block and the attribute information.

[0208] In addition to providing the above-mentioned method and apparatus for data migration, the present disclosure also provides equipment, medium and products for data migration. The following will be described in conjunction with the accompanying drawings. The above description of the method and apparatus also applies to the equipment, medium and products to be described below, unless otherwise explicitly stated.

[0209] Fig.13 is a block diagram illustrating a device 1300 for data migration according to an embodiment of the present disclosure.

[0210] See also Fig.13 , the device 1300 may include a processor 1301 and a memory 1302. The processor 1301 and the memory 1302 may be connected via a bus 1303.

[0211] The processor 1301 can perform various actions and processes according to the program stored in the memory 1302. Specifically, the processor 1301 can be an integrated circuit chip with signal processing capabilities. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can be an X86 architecture or an ARM architecture.

[0212] The memory 1302 stores computer executable instructions, and the above-mentioned method for data migration is implemented when the computer executable instructions are executed by the processor 1301. The memory 1302 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting explanation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM). It should be noted that the memory of the method described herein is intended to include but is not limited to these and any other suitable types of memory.

[0213] Fig.14 A schematic diagram showing a recording medium according to the present disclosure is shown.

[0214] like Fig.14As shown, a computer executable instruction 1410 is stored on the recording medium 1400. When the computer executable instruction 1410 is executed by a processor, the method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The recording medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0215] The computer program product provided by the present disclosure may store computer executable instructions. When the computer executable instructions are executed by a processor, the processor is prompted to execute the method according to the embodiment of the present disclosure described above with reference to the above drawings.

[0216] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0217] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0218] The exemplary embodiments of the present invention described in detail above are merely illustrative, rather than restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present invention, and such modifications should fall within the scope of the present invention.

[0219] In the above specific embodiments, it can be seen that different features are combined together in the examples. This disclosure should not be understood as an intention that the example clauses have more features than the features explicitly mentioned in each clause. On the contrary, various aspects of the present disclosure may include fewer features than all the features of the disclosed single example clauses. Therefore, the following clauses should be considered to be included in the specification, where each clause itself can be used as a separate example. Although each subordinate clause can refer to a specific combination with one of the other clauses in these clauses, the aspects of the subordinate clauses are not limited to specific combinations. It should be understood that other example clauses may also include a combination of the subject matter of the subordinate clauses with any other subordinate clauses or independent clauses, or any combination of features with other subordinate clauses and independent clauses. Various aspects disclosed herein explicitly include these combinations, unless it is explicitly expressed or it can be easily inferred that a specific combination is not intended (for example, contradictory aspects, such as defining an element as both an insulator and a conductor). In addition, it is also intended that various aspects of the clauses may also be included in any other independent clauses, even if the clause is not directly subordinate to the independent clause.

[0220] The following numbered clauses describe implementation examples:

[0221] Clause 1. A method for data migration, comprising: obtaining a first data block to be migrated and attribute information corresponding to the first data block to be migrated, wherein when the first data block to be migrated is stored in a first storage block, the attribute information is stored in a second storage block, and the first storage block corresponds to the second storage block; migrating the first data block to be migrated and the attribute information to a third storage block and a fourth storage block respectively; and updating the metadata information of the first data block to be migrated based on the third storage block and the attribute information.

[0222] Clause 2. The method according to clause 1, wherein the attribute information includes identification information of a data object corresponding to the first data block to be migrated and identification information of the first data block to be migrated.

[0223] Clause 3. A method according to any of the above clauses, wherein the updating of the metadata information of the first data block to be migrated based on the third storage block and the attribute information comprises: updating the storage location information indicating the first data block to be migrated in the metadata information based on the third storage block; and obtaining the updated metadata information based on the attribute information and the storage location information.

[0224] Clause 4. The method according to any of the above clauses further comprises: storing the first to-be-migrated data block and the attribute information in the first data block and the second data block respectively in a parallel manner.

[0225] Clause 5. The method according to any of the above clauses further includes: when the memory space occupied by the total attribute information corresponding to multiple data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size, each attribute information in the total attribute information including the attribute information is stored in a corresponding storage block respectively.

[0226] Clause 6. The method according to any of the above clauses also includes: storing the attribute information in a fifth storage block, loading the attribute information from the fifth storage block into the memory space; and determining whether the memory space occupied by the total attribute information corresponding to multiple data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size.

[0227] Clause 7. A device for data migration, comprising: an acquisition unit, which is configured to acquire a first data block to be migrated and attribute information corresponding to the first data block to be migrated, wherein when the first data block to be migrated is stored in a first storage block, the attribute information is stored in a second storage block, and the first storage block corresponds to the second storage block; a migration unit, which is configured to migrate the first data block to be migrated and the attribute information to a third storage block and a fourth storage block respectively; and an update unit, which is configured to update the metadata information of the first data block to be migrated based on the third storage block and the attribute information.

[0228] Clause 8. The apparatus according to Clause 7, wherein the attribute information includes identification information of a data object corresponding to the first data block to be migrated and identification information of the first data block to be migrated.

[0229] Clause 9. The apparatus according to any of the preceding clauses, wherein the updating unit comprises:

[0230] The first updating subunit is configured to update the storage location information indicating the first data block to be migrated in the metadata information based on the third storage block; the second updating subunit is configured to obtain the updated metadata information based on the attribute information and the storage location information.

[0231] Clause 10. The apparatus according to any of the above clauses further comprises: a first storage unit configured to: store the first to-be-migrated data block and the attribute information in the first data block and the second data block respectively in a parallel manner.

[0232] Clause 11. The device according to any of the above clauses further includes: a second storage unit, which is configured to: when the memory space occupied by the total attribute information corresponding to multiple data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size, store each attribute information in the total attribute information including the attribute information in the corresponding storage block respectively.

[0233] Clause 12. The device according to any of the above clauses further includes: a third storage unit, which is configured to: store the attribute information in a fifth storage block; a loading unit, which is configured to load the attribute information from the fifth storage block into the memory space; and a judgment unit, which is configured to judge whether the memory space occupied by the total attribute information corresponding to multiple data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size.

[0234] Clause 13. A device for data migration, comprising: a component for obtaining a first data block to be migrated and attribute information corresponding to the first data block to be migrated, wherein when the first data block to be migrated is stored in a first storage block, the attribute information is stored in a second storage block, and the first storage block corresponds to the second storage block; a component for migrating the first data block to be migrated and the attribute information to a third storage block and a fourth storage block respectively; and a component for updating metadata information of the first data block to be migrated based on the third storage block and the attribute information.

[0235] Clause 14. A device for data migration, comprising: a processor, and a memory, wherein the memory stores computer executable instructions, and when the computer executable instructions are executed by the processor, the processor is prompted to perform the method as described in any one of the above clauses.

[0236] Clause 15. A computer-readable recording medium storing computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor is prompted to perform the method as described in any one of the above clauses.

[0237] Clause 16. A computer program product storing computer executable instructions, wherein the computer executable instructions, when executed by a processor, cause the processor to perform the method as described in any one of the above.

Claims

1. A method for data migration, comprising: Acquire a first data block to be migrated and attribute information corresponding to the first data block to be migrated, wherein the attribute information includes identification information of a data object corresponding to the first data block to be migrated and identification information of the first data block to be migrated, wherein in the case where the first data block to be migrated is stored in a first storage block, the attribute information is stored in a second storage block, and the first storage block corresponds to the second storage block; Migrating the first data block to be migrated and the attribute information to a third storage block and a fourth storage block respectively; and The metadata information of the first data block to be migrated is updated based on the third storage block and the attribute information.

2. The method according to claim 1, wherein: The updating of metadata information of the first data block to be migrated based on the third storage block and the attribute information includes: Based on the third storage block, updating the storage location information indicating the first data block to be migrated in the metadata information; Based on the attribute information and the storage location information, the updated metadata information is obtained.

3. The method according to claim 1, further comprising: The first data block to be migrated and the attribute information are stored in the first storage block and the second storage block respectively in a parallel manner.

4. The method according to claim 1, further comprising: When the memory space occupied by the total attribute information corresponding to the plurality of data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size, each attribute information in the total attribute information including the attribute information is stored in a corresponding storage block.

5. The method according to claim 4, further comprising: storing the attribute information in a fifth storage block, Loading the attribute information from the fifth storage block into the memory space; It is determined whether the total memory space occupied by the attribute information corresponding to the plurality of data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size.

6. A device for data migration, comprising: an acquiring unit, configured to acquire a first data block to be migrated and attribute information corresponding to the first data block to be migrated, wherein the attribute information includes identification information of a data object corresponding to the first data block to be migrated and identification information of the first data block to be migrated, wherein in the case where the first data block to be migrated is stored in a first storage block, the attribute information is stored in a second storage block, and the first storage block corresponds to the second storage block; a migration unit configured to migrate the first data block to be migrated and the attribute information to a third storage block and a fourth storage block respectively; and An updating unit is configured to update metadata information of the first data block to be migrated based on the third storage block and the attribute information.

7. The device according to claim 6, wherein: The updating unit comprises: a first updating subunit, configured to update storage location information indicating the first to-be-migrated data block in the metadata information based on the third storage block; The second updating subunit is configured to obtain the updated metadata information based on the attribute information and the storage location information.

8. The apparatus according to claim 6, further comprising: The first storage unit is configured to store the first to-be-migrated data block and the attribute information in parallel in the first storage block and the second storage block respectively.

9. The apparatus according to claim 6, further comprising: The second storage unit is configured as: When the memory space occupied by the total attribute information corresponding to the plurality of data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size, each attribute information in the total attribute information including the attribute information is stored in a corresponding storage block.

10. The apparatus according to claim 9, further comprising: A third storage unit, configured to: store the attribute information in a fifth storage block; a loading unit configured to load the attribute information from the fifth storage block into a memory space; The judging unit is configured to judge whether the total memory space occupied by the total attribute information corresponding to the plurality of data blocks to be migrated including the first data block to be migrated reaches a predetermined threshold size.

11. A device for data migration, comprising: a component for acquiring a first data block to be migrated and attribute information corresponding to the first data block to be migrated, wherein the attribute information includes identification information of a data object corresponding to the first data block to be migrated and identification information of the first data block to be migrated, wherein in the case where the first data block to be migrated is stored in a first storage block, the attribute information is stored in a second storage block, and the first storage block corresponds to the second storage block; A component for migrating the first data block to be migrated and the attribute information to a third storage block and a fourth storage block respectively; and A component for updating metadata information of the first data block to be migrated based on the third storage block and the attribute information.

12. A device for data migration, comprising: processor, and A memory storing computer executable instructions, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 5.

13. A computer-readable recording medium storing computer-executable instructions, wherein: The computer executable instructions, when executed by a processor, cause the processor to perform the method of any one of claims 1-5.

14. A computer program product storing computer executable instructions, wherein: The computer executable instructions, when executed by a processor, cause the processor to perform the method of any one of claims 1-5.

Images