Data migration method and device, storage medium and electronic device

Abstract

The present disclosure provides a data migration method and device, a storage medium and an electronic device, the data migration method can start the data migration verification operation of the second index disk after migrating the historical object metadata in the first index disk to the second index disk, and then migrate the process object metadata to the second index disk. Since the data migration verification operation of the second index disk is not started during the migration of the historical object metadata, the historical object metadata can be directly migrated to the second index disk, and the migration of the historical object metadata based on the data migration verification operation is not required, so that the migration efficiency of the data can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data migration method and apparatus, storage medium and electronic device. Background Technology

[0002] In object storage, a file is treated as an object, its content as object data, and its attributes as object metadata. Typically, object data and object metadata are stored on different disks. The disk storing the object metadata is called the index disk. In practice, when the existing index disk's storage space is insufficient, a new index disk can be added. To ensure a relatively balanced distribution of data across the index disks, some object metadata from the existing index disks is migrated to the newly added index disk.

[0003] In current implementations, during data migration, data is migrated from the existing index disk to the new index disk, and newly written data to the existing index disk is simultaneously written to the new index disk. To avoid duplicate storage of the same object's metadata on the new index disk, a data migration verification operation is typically performed on the new index disk to write each received object metadata record to the new index disk. This data migration verification operation means: if it is determined that the object metadata record exists on the new index disk, and the version number of the object metadata record is not the latest, then the object metadata record is not written; if it is determined that the object metadata record exists on the new index disk, and the version number of the object metadata record is the latest, then the object metadata record is written; if it is determined that the object metadata record does not exist on the new index disk, then the object metadata record is written.

[0004] However, existing technologies have low data migration efficiency. Summary of the Invention

[0005] In view of the above, embodiments of this disclosure provide a data migration method and apparatus, a storage medium and an electronic device.

[0006] According to a first aspect of this disclosure, a data migration method applied to a first node is proposed, the method comprising:

[0007] In the first index disk, determine the regular storage area to be migrated, and create a first custom storage area and a second custom storage area associated with the regular storage area;

[0008] Historical object metadata is written to the first custom storage area. The historical object metadata is the object metadata written to the regular storage area before the data migration begins.

[0009] migrating the historical object metadata in the first custom storage area to a second index disk, and writing process object metadata into the second custom storage area, the process object metadata being object metadata written into the regular storage area in the process of migrating the historical object metadata;

[0010] in response to completing migrating the historical object metadata, migrating the process object metadata in the second custom storage area to the second index disk;

[0011] wherein the second index disk is managed by a second node, and the second node, in response to completing migrating the historical object metadata, synchronously receives newly added object metadata with the first node and starts a data migration verification operation of the second index disk.

[0012] In combination with any of the embodiments provided by the present disclosure,

[0013] The writing of the historical object metadata into the first custom storage area includes:

[0014] writing keys of the historical object metadata into the first custom storage area;

[0015] Correspondingly, the migrating of the historical object metadata in the first custom storage area to the second index disk includes:

[0016] obtaining corresponding values from the regular storage area according to the keys of the historical object metadata stored in the first custom storage area, to obtain complete historical object metadata;

[0017] migrating the historical object metadata to the second index disk;

[0018] and / or

[0019] The writing of the process object metadata into the second custom storage area includes:

[0020] writing keys of the process object metadata into the second custom storage area;

[0021] Correspondingly, the migrating of the process object metadata in the second custom storage area to the second index disk includes:

[0022] obtaining corresponding values from the regular storage area according to the keys of the process object metadata stored in the second custom storage area, to obtain complete process object metadata;

[0023] migrating the process object metadata to the second index disk.

[0024] In combination with any embodiment provided by the present disclosure,

[0025] The migrating the historical object metadata in the first custom storage area to a second index disk comprises:

[0026] sequentially scanning each historical object metadata in the first custom storage area, in response to the number of scanned historical object metadata reaching a preset first threshold, constructing a data migration request containing the scanned preset first threshold number of historical object metadata, and initiating the data migration request to the second node, so that the second node writes the historical object metadata contained in each data migration request received to the second index disk;

[0027] and / or

[0028] The migrating the process object metadata in the second custom storage area to the second index disk comprises:

[0029] sequentially scanning each process object metadata in the second custom storage area, in response to the number of scanned process object metadata reaching a preset first threshold, constructing a data migration request containing the scanned preset first threshold number of process object metadata, and initiating the data migration request to the second node, so that the second node writes the process object metadata contained in each data migration request received to the second index disk.

[0030] In combination with any embodiment provided by the present disclosure, the initiating the data migration request to the second node comprises:

[0031] sending the data migration request to a data migration management module, so that the data migration management module, upon receiving the data migration request, if it is determined that the number of data migration requests in a sending queue does not reach a preset second threshold, adds the data migration request to the sending queue and notifies the first node to continue scanning object metadata, so that the data migration management module sequentially sends each data migration request in the sending queue to the second node.

[0032] In combination with any embodiment provided by the present disclosure, if the data migration management module determines that the number of data migration requests in the sending queue has reached the preset second threshold, it caches the data migration request, and in the case that the number of data migration requests in the sending queue is less than a preset third threshold, adds the cached data migration request to the sending queue; the preset third threshold is less than or equal to the preset second threshold.

[0033] According to a second aspect of the present disclosure, a data migration method applied to a second node is provided, wherein a first node managing a first index disk of an object storage performs: determining a regular storage area to be migrated in the first index disk, and creating a first custom storage area and a second custom storage area associated with the regular storage area; writing historical object metadata in the first custom storage area, the historical object metadata being object metadata written in the regular storage area before data migration starts; in response to a data migration instruction, migrating the historical object metadata in the first custom storage area to a second index disk, and writing process object metadata in the second custom storage area, the process object metadata being object metadata written in the regular storage area during migration of the historical object metadata; in response to completion of migration of the historical object metadata, migrating the process object metadata in the second custom storage area to the second index disk; the method comprises:

[0034] writing the received historical object metadata into the second index disk to complete migration of the historical object metadata;

[0035] in response to completion of migration of the historical object metadata, synchronously receiving newly added object metadata with the first node, and starting a data migration verification operation of the second index disk;

[0036] writing the newly added object metadata into the second index disk, and writing the received process object metadata into the second index disk based on the data migration verification operation.

[0037] According to any one of the embodiments provided by the present disclosure,

[0038] the writing of the received historical object metadata into the second index disk comprises:

[0039] writing the received historical object metadata into an SST file, and inserting the SST file containing the historical object metadata into the second index disk through an SST file insertion interface of the second index disk;

[0040] or

[0041] the synchronously receiving of the newly added object metadata comprises:

[0042] writing the received newly added object metadata into an SST file, and inserting the SST file containing the newly added object metadata into the second index disk through an SST file insertion interface of the second index disk;

[0043] or

[0044] The writing of the received process object metadata into the second index disk comprises:

[0045] The received process object metadata is written into an SST file, and an SST file containing the process object metadata is inserted into the second index disk through an SST file insertion interface of the second index disk.

[0046] According to a third aspect of the present disclosure, a data migration method is provided, which comprises:

[0047] In a first index disk of object storage, a regular storage area to be subjected to data migration is determined, and a first custom storage area and a second custom storage area associated with the regular storage area are created;

[0048] Historical object metadata is written into the first custom storage area, the historical object metadata being object metadata written into the regular storage area before the start of data migration;

[0049] In response to a data migration instruction, the historical object metadata in the first custom storage area is migrated to a second index disk, and process object metadata is written into the second custom storage area, the process object metadata being object metadata written into the regular storage area in the process of migrating the historical object metadata;

[0050] In response to completion of migrating the historical object metadata, new object metadata is synchronously written into the regular storage area and the second index disk, and after starting a data migration verification operation of the second index disk, the process object metadata in the second custom storage area is migrated to the second index disk.

[0051] According to a fourth aspect of the present disclosure, a data migration device applied to a first node is provided, which comprises:

[0052] A creating module is configured to determine, in the first index disk, a regular storage area to be subjected to data migration, and create a first custom storage area and a second custom storage area associated with the regular storage area;

[0053] A first writing module is configured to write historical object metadata into the first custom storage area, the historical object metadata being object metadata written into the regular storage area before the start of data migration;

[0054] a first migration module, configured to, in response to a data migration instruction, migrate the historical object metadata in the first customized storage area to a second index disk, and write process object metadata into the second customized storage area, the process object metadata being object metadata written in the regular storage area in a process of migrating the historical object metadata;

[0055] a second migration module, configured to, in response to completion of migrating the historical object metadata, migrate the process object metadata in the second customized storage area to the second index disk;

[0056] wherein the second index disk is managed by a second node, and the second node is configured to, in response to completion of migrating the historical object metadata, synchronously receive new object metadata from the first node, and start a data migration verification operation of the second index disk.

[0057] According to a fifth aspect of the present disclosure, a data migration apparatus applied to a second node is provided, and the apparatus comprises:

[0058] a second writing module, configured to write the received historical object metadata into the second index disk to complete migration of the historical object metadata;

[0059] a receiving module, configured to, in response to completion of migrating the historical object metadata, synchronously receive new object metadata from the first node, and start a data migration verification operation of the second index disk;

[0060] a third writing module, configured to write the new object metadata into the second index disk, and write received process object metadata into the second index disk based on the data migration verification operation.

[0061] According to a sixth aspect of the present disclosure, a data migration apparatus is provided, and the apparatus comprises:

[0062] a determining module, configured to determine a regular storage area to be subjected to data migration in a first index disk of object storage, and create a first customized storage area and a second customized storage area associated with the regular storage area;

[0063] a fourth writing module, configured to write historical object metadata into the first customized storage area, the historical object metadata being object metadata written in the regular storage area before data migration starts;

[0064] a third migration module configured to, in response to a data migration instruction, migrate the historical object metadata in the first custom storage area to a second index disk, and write process object metadata in the second custom storage area, the process object metadata being object metadata written in the regular storage area during migration of the historical object metadata;

[0065] a fifth writing module configured to, in response to completion of migration of the historical object metadata, synchronously write new object metadata to the regular storage area and the second index disk, and after starting a data migration verification operation of the second index disk, migrate the process object metadata in the second custom storage area to the second index disk.

[0066] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided, the computer-readable storage medium storing machine-readable instructions, the machine-readable instructions, when invoked and executed by a processor, causing the processor to implement the data migration method of any embodiment of the present disclosure.

[0067] According to an eighth aspect of the present disclosure, an electronic device is provided, comprising

[0068] a processor;

[0069] a memory for storing processor-executable instructions;

[0070] The processor is configured to execute the data migration method of any embodiment of the present disclosure.

[0071] The technical solutions provided by the embodiments of the present disclosure can include the following beneficial effects:

[0072] The data migration method and device, storage medium and electronic device provided by the embodiments of the present disclosure can start a data migration verification operation of the second index disk after migrating historical object metadata in the first index disk to the second index disk, and then migrate process object metadata to the second index disk. Since the data migration verification operation of the second index disk is not started during migration of the historical object metadata, the historical object metadata can be directly migrated to the second index disk, and the migration of historical object metadata based on the data migration verification operation is not required, which can improve the efficiency of data migration.

[0073] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0074] The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure.

[0075] Figure 1 is a flow chart of a data migration method according to an exemplary embodiment of the present disclosure;

[0076] Figure 2 is a flow chart of another data migration method according to an exemplary embodiment of the present disclosure;

[0077] Figure 3 is a flow chart of another data migration method according to an exemplary embodiment of the present disclosure;

[0078] Figure 4 is a structural schematic diagram of a data migration apparatus according to an exemplary embodiment of the present disclosure;

[0079] Figure 5 is a structural schematic diagram of another data migration apparatus according to an exemplary embodiment of the present disclosure;

[0080] Figure 6 is a structural schematic diagram of another data migration apparatus according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0081] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description of the exemplary embodiments is intended to apply to various alternative embodiments of the present disclosure. The following description is not limited to the exemplary embodiments but is intended to apply more broadly. It should be noted that the description of the exemplary embodiments is intended solely for the purpose of illustrating the principles of the present disclosure and should not be interpreted as limiting the scope of the present disclosure.

[0082] The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used in the present disclosure and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0083] It should be understood that, although the terms first, second, third, etc. can be employed in this disclosure to describe various information, the information should not be limited to these terms. These terms are only used to differentiate one piece of information from another piece of information of the same type. For example, without departing from the scope of the disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if' as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0084] In object storage, a file is an object, the content of the file is object data, and the attribute of the file is object metadata. The object data and the object metadata are usually stored on different disks, the disk storing the object data is referred to as a data disk, and the disk storing the object metadata is referred to as an index disk. When an object is uploaded, storage space is allocated for the object on the data disk, and then the object data of the object is written into the allocated space on the data disk, and after the object data is written, the object metadata of the object is written to the corresponding index disk. In actual application, when the storage space of an existing index disk is insufficient, a new index disk can be added. In order to balance the amount of data stored on each index disk, part of the object metadata on the existing index disk is migrated to the new index disk.

[0085] In existing implementation, when data is migrated, the data on the existing index disk is migrated to the new index disk, and the data newly written to the existing index disk is synchronously written to the new index disk. In order to avoid the situation that the object metadata of the same object is repeatedly stored on the new index disk, each piece of object metadata received is usually written to the new index disk based on a data migration verification operation of the new index disk, where the data migration verification operation refers to: if it is determined that the piece of object metadata exists on the new index disk, and the version number of the piece of object metadata is not the latest, the piece of object metadata is not written; if it is determined that the piece of object metadata exists on the new index disk, but the version number of the piece of object metadata is the latest, the piece of object metadata is written; if it is determined that the piece of object metadata does not exist on the new index disk, the piece of object metadata is written. In the related field, the above-mentioned data migration verification operation is also referred to as a get operation.

[0086] That is, in the existing data migration method, each piece of data written to the new index disk needs to be verified, so that the existing data migration method is low in efficiency.

[0087] Therefore, embodiments of the present disclosure provide a data migration method, in which the data to be written to the new index disk can be divided into multiple parts for migration, and part of the data does not need to be migrated based on the data verification operation of the new index disk, so that the migration efficiency of the data can be improved.

[0088] The data migration method of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0089] Figure 1 is a flowchart of a data migration method according to an exemplary embodiment of the present disclosure, which is applied to a first node managing a first index disk of object storage, i.e., the existing index disk described above. As shown in Figure 1 the exemplary embodiment method can include the following steps:

[0090] In step 100, a regular storage area to be migrated is determined in the first index disk, and a first custom storage area and a second custom storage area associated with the regular storage area are created.

[0091] In practical applications, the regular storage area can be referred to as Default CF, the first custom storage area newly created as CheckPoint, and the second custom storage area newly created as Migrate CF.

[0092] In step 102, historical object metadata is written into the first custom storage area.

[0093] The historical object metadata is the object metadata written into the regular storage area before the start of data migration.

[0094] After the CheckPoint and Migrate CF associated with Default CF are created, the historical object metadata in the Default CF can be written into the created CheckPoint first.

[0095] In step 104, in response to a data migration instruction, the historical object metadata in the first custom storage area is migrated to a second index disk, and process object metadata is written into the second custom storage area.

[0096] The process object metadata is the object metadata written into the regular storage area during the migration of the historical object metadata.

[0097] In the example, the first node can receive the data migration instruction sent by the service node. After receiving the data migration instruction, the first node can migrate the historical object metadata stored in the CheckPoint to the second index disk, i.e., the newly added index disk. While migrating the historical object metadata to the second index disk, the first node can also synchronize the process object metadata written in the Default CF during the migration of the historical object metadata to the Migrate CF. That is, the data stored in the Migrate CF is the data newly written in the Default CF during the migration of the historical object metadata.

[0098] In step 106, in response to completing the migration of the historical object metadata, the process object metadata in the second custom storage area is migrated to the second index disk.

[0099] After migrating the historical object metadata stored in the CheckPoint to the second index disk, the first node can further migrate the data stored in the Migrate CF to the second index disk.

[0100] The second index disk is managed by a second node. In response to completing the migration of the historical object metadata, the second node synchronously receives the newly added object metadata from the first node and starts the data migration verification operation of the second index disk. Details of the operation of the second node managing the second index disk will be described in subsequent embodiments, and will not be described here.

[0101] In the data migration method provided by the embodiments of the present disclosure, the data migration verification operation of the second index disk can be started after the historical object metadata in the first index disk is migrated to the second index disk, and then the process object metadata is migrated to the second index disk. Since the data migration verification operation of the second index disk is not started during the migration of the historical object metadata, the historical object metadata can be directly migrated to the second index disk, without the need to migrate the historical object metadata based on the data migration verification operation, which can improve the data migration efficiency.

[0102] The following further describes some steps in the data migration method described above.

[0103] In the above embodiments, the Default CF stores the full amount of data, the CheckPoint and the Migrate CF store part of the data stored in the Default CF respectively, and the data are in the form of key-value pairs. In order to reduce the space occupied by the stored data, the present example proposes the following solutions:

[0104] Solution one: the first node only writes the key of the historical object metadata into the CheckPoint when writing the historical object metadata into the CheckPoint.

[0105] Correspondingly, when migrating the historical object metadata in the CheckPoint to the second index disk, the corresponding value is first obtained from the Default CF according to the key of the historical object metadata stored in the CheckPoint to obtain the complete historical object metadata, and then the complete historical object metadata is migrated to the second index disk.

[0106] Solution two: the first node only writes the key of the process object metadata into the Migrate CF when writing the process object metadata into the Migrate CF.

[0107] Correspondingly, when migrating the process object metadata in the Migrate CF to the second index disk, the corresponding value is first obtained from the Default CF according to the key of the process object metadata stored in the Migrate CF to obtain the complete process object metadata, and then the complete process object metadata is migrated to the second index disk.

[0108] It should be noted that in actual application, only solution one can be applied in the above data migration method, or only solution two can be applied in the above data migration method. Alternatively, both solution one and solution two can be applied in the above data migration method, and the present disclosure does not limit this.

[0109] In the data migration method provided by the embodiments of the present disclosure, only the key of the historical object metadata is written when writing the historical object metadata into the CheckPoint, and only the key of the process object metadata is written when writing the process object metadata into the Migrate CF, thereby reducing the memory occupied by the historical object metadata and the process object metadata.

[0110] In the above embodiments, when migrating the historical object metadata (or the process object metadata) in the CheckPoint (or the Migrate CF) to the second index disk, the first node can build a data migration request containing each historical object metadata (or process object metadata) scanned from the CheckPoint (or the Migrate CF) and initiate the data migration request to the second node. In order to reduce the number of data migration requests initiated by the first node, the present example proposes the following solutions:

[0111] Solution one: the first node scans each historical object metadata in the CheckPoint in turn, and when the number of historical object metadata scanned reaches a preset first threshold, builds a data migration request containing the preset first threshold number of historical object metadata scanned, and initiates the data migration request to the second node, so that the second node writes the historical object metadata contained in the data migration request to the second index disk according to the received data migration request.

[0112] Optionally, the first node can save each historical object metadata scanned in a list, and then continue to scan the next historical object metadata. When the number of historical object metadata in the list reaches a preset first threshold, such as 1000, a data migration request containing the above 1000 historical object metadata is built, and the data migration request is initiated to the second node, so that the second node migrates the 1000 historical object metadata in the list according to the data migration request.

[0113] Optionally, after the first node scans all the historical object metadata in the CheckPoint, it can further determine whether there is data in the list. If there is data, whether the number of data reaches the above 1000 or not, an additional data migration request containing all data in the list is built, and the data migration request is initiated to the second node. If there is no data, no additional operation is performed.

[0114] Solution two: the first node scans each process object metadata in the Migrate CF in turn, and when the number of process object metadata scanned reaches a preset first threshold, builds a data migration request containing the preset first threshold number of process object metadata scanned, and initiates the data migration request to the second node, so that the second node writes the process object metadata contained in the data migration request to the second index disk according to the received data migration request.

[0115] Optionally, the first node can save the process object metadata obtained by scanning in a list, and then continue to scan the next process object metadata. When the number of process object metadata in the list reaches a preset first threshold, for example, reaches 1000, a data migration request containing the above 1000 process object metadata is constructed, and the data migration request is initiated to the second node, so that the second node migrates the 1000 process object metadata in the list according to the data migration request.

[0116] Optionally, after the first node scans all the process object metadata in the Migrate CF, it can further determine whether there is data in the list. If there is data, whether the number of data reaches the above 1000 or not, an additional data migration request containing all data in the list is constructed, and the data migration request is initiated to the second node. If there is no data, no additional operation is performed.

[0117] In the data migration method provided by the embodiments of the present disclosure, when the number of scanned metadata reaches a first preset threshold, a data migration request containing the metadata of the first preset threshold is constructed, and the data migration request is initiated to the second node when the metadata is migrated to the second index disk. Thus, the number of data migration requests initiated by the first node is reduced, and the power consumption is reduced.

[0118] In the above embodiment, after the first node initiates the data migration request containing the first preset threshold number of data, it needs to receive the response message sent by the second node, that is, after the second node migrates the data in the data migration message to the second index disk, the first node can continue to scan and obtain the next data, so that the data migration efficiency is low. In order to further improve the data migration efficiency, the present example proposes the following scheme:

[0119] The first node sends the data migration request to the data migration management module. When the data migration management module receives the data migration request, if it is determined that the number of data migration requests in the sending queue does not reach a preset second threshold, the data migration request is added to the sending queue, and each data migration request in the sending queue is sent to the second node in turn. Moreover, after receiving the data migration request, the data migration management module can notify the first node to continue to scan the object metadata.

[0120] For example, the preset second threshold value can be 16. When the data migration management module receives the data migration request sent by the first node, the data migration management module can determine whether the number of data migration requests in the sending queue reaches 16. If the number of data migration requests does not reach 16, the data migration management module can add the data migration request to the sending queue. At the same time, the data migration management module can inform the first node to continue scanning the object metadata.

[0121] In the data migration method provided by the embodiments of the present disclosure, the data migration management module can control the sending queue to store a plurality of data migration requests. When the scanning speed of the first node is faster than the migration speed of the second node, the data migration management module can store the excess data migration requests in the sending queue, so as to prevent the situation that the first node needs to wait for scanning the object metadata after the migration speed of the second node suddenly increases. At the same time, the data migration management module can add the data migration request message to the sending queue and inform the first node to continue scanning the object metadata, so as to improve the data migration efficiency.

[0122] In an optional example, if the data migration management module determines that the number of data migration requests in the sending queue reaches the preset second threshold value, the data migration management module can cache the data migration request. In addition, if the number of data migration requests in the sending queue is less than a preset third threshold value, the data migration management module can add the cached data migration request to the sending queue.

[0123] In the preset third threshold value is less than or equal to the preset second threshold value.

[0124] For example, the second preset threshold value is 16. At this time, the third preset threshold value can be 10 or 8, and the present disclosure does not limit the third preset threshold value. For the convenience of description, the third preset threshold value is 8 in the following embodiments.

[0125] When the data migration management module determines that the number of data migration requests in the sending queue reaches 16, the data migration management module can store the data migration request in a preset cache pool. When the number of data migration requests in the sending queue is less than 8, the data migration management module can add the data migration request in the cache pool to the sending queue. At the same time, the data migration management module can inform the first node to continue scanning the object metadata.

[0126] The data migration method provided by the embodiments of the present disclosure can cache the currently received data migration request when the data migration management module determines that the number of data migration requests in the sending queue reaches the second preset threshold, and add the cached data migration request to the sending queue when the number of data migration requests in the sending queue is less than a third preset threshold, so as to ensure that there are enough data migration requests in the sending queue for the second node to migrate, prevent the second node from waiting for the first node to scan the object metadata, and improve the data migration efficiency.

[0127] Figure 2 is a flowchart of a data migration method according to an exemplary embodiment of the present disclosure, which is applied to a second node that manages a second index disk of object storage, that is, the above-mentioned newly added index disk. As shown in Figure 2 the exemplary embodiment method can include the following steps:

[0128] In step 200, write the received historical object metadata to the second index disk to complete migration of the historical object metadata.

[0129] In step 202, in response to completing migration of the historical object metadata, synchronize with the first node to receive newly added object metadata, and start a data migration verification operation of the second index disk.

[0130] The newly added object metadata is object metadata written to the Default CF in the process of migrating the process object metadata.

[0131] In step 204, write the newly added object metadata to the second index disk, and based on the data migration verification operation, write the received process object metadata to the second index disk.

[0132] The second node can start the data migration verification operation of the second index disk after completing migration of the historical object metadata, and then write the process object metadata and the newly added object metadata to the second index disk.

[0133] In a specific implementation, each piece of object metadata has a version number corresponding thereto, which corresponds to the uploading time. The later the uploading time, the newer the version number. After the data migration verification operation of the second index disk is started, when a piece of object metadata is written into the second index disk, it is necessary to first determine whether the metadata of the object exists on the second index disk. If not, the piece of object metadata is written into the second index disk. If yes, it is further necessary to determine whether the version number of the piece of object metadata is the latest. If yes, the piece of object metadata is written into the second index disk, and the object metadata with an older version number stored in the second index disk is overwritten. Otherwise, the piece of object metadata is not written into the second index disk.

[0134] In the above embodiment, when the second node writes object metadata into the second index disk, the second node needs to sequentially write the object metadata into a logfile, a memtable and an sstfile based on a put interface of the second index disk. Specifically, when data is written, the logfile is written first, and then the memtable in the memory is written. After a memtable is written full, a background thread writes the data in the memtable to the disk to generate an sstfile. The sstfile is a data persistence file. The above data writing process is relatively complex, and the data migration efficiency is low. In order to further improve the data migration efficiency, the present example proposes the following solutions.

[0135] Solution one: When the second node writes the received historical object metadata into the second index disk, the second node first writes the received historical object metadata into an SST file, and then directly inserts the SST file containing the historical object metadata into the second index disk through an SST file insertion interface of the second index disk.

[0136] Optionally, the SST file insertion interface can be an ingestExternalFile interface.

[0137] Solution two: When the second node writes the received new object metadata into the second index disk, the second node first writes the received new object metadata into an SST file, and then directly inserts the SST file containing the new object metadata into the second index disk through an ingestExternalFile interface of the second index disk.

[0138] Solution three: When the second node writes the received process object metadata into the second index disk, the second node first writes the received process object metadata into an SST file, and then directly inserts the SST file containing the process object metadata into the second index disk through an ingestExternalFile interface of the second index disk.

[0139] In an optional example, when writing the received object metadata into the SST file, it can be determined according to the name of the received object metadata whether there is an SST file with the same name.

[0140] When it is determined that there is, it indicates that the object metadata corresponding to the SST file has been received before, and the object metadata can be directly written into the corresponding SST file.

[0141] When it is determined that there is not, the SST file with the corresponding name can be created first, and then the object metadata is written into the newly created SST file.

[0142] It should be noted that any of the above schemes can be applied to the data migration method provided by the embodiments of the present disclosure to improve the data migration efficiency. Alternatively, the three schemes described above can also be applied to the data migration method provided by the embodiments of the present disclosure, so as to maximize the data migration efficiency.

[0143] In the data migration method provided by the embodiments of the present disclosure, the second node can directly write the object metadata in the SST file, and then directly insert the SST file into the second index disk through the ingestExternalFile interface of the second index disk, without writing into the logfile and the memtable, so as to improve the data migration efficiency.

[0144] Figure 3 It is a flowchart of a data migration method according to an example embodiment, which is executed by a server. Specifically, the method can be executed by a service node on the server. The service node can control the first node and the second node described above. As shown in Figure 3 The example embodiment method can include the following steps:

[0145] In step 300, in the first index disk of the object storage, a regular storage area to be migrated is determined, and a first custom storage area and a second custom storage area associated with the regular storage area are created.

[0146] In step 302, historical object metadata is written into the first custom storage area.

[0147] The historical object metadata is object metadata written into the regular storage area before the data migration starts.

[0148] In step 304, in response to a data migration instruction, the historical object metadata in the first custom storage area is migrated to a second index disk, and process object metadata is written into the second custom storage area.

[0149] The process object metadata is object metadata written in the regular storage area during migration of the historical object metadata.

[0150] In step 306, in response to completion of migration of the historical object metadata, newly added object metadata is synchronously written into the regular storage area and the second index disk, and the process object metadata in the second custom storage area is migrated to the second index disk after starting the data migration verification operation of the second index disk.

[0151] In the data migration method provided by the embodiments of the present disclosure, the data to be migrated can be divided into three parts, namely historical object metadata, process object metadata and newly added object metadata. For the historical object metadata, migration can be performed before starting the data migration verification operation of the second index disk, that is, the historical object metadata can be directly written into the second index disk, without the need to perform the data migration verification operation, thereby improving the data migration efficiency.

[0152] For each of the foregoing method embodiments, in order to simply describe, it is expressed as a combination of a series of actions, but those skilled in the art should know that the present disclosure is not limited by the order of the described actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously.

[0153] Secondly, those skilled in the art should know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0154] Corresponding to the foregoing application function implementation method embodiments, the present disclosure also provides application function implementation device and corresponding terminal embodiments.

[0155] Figure 4 FIG. 1 is a structural schematic diagram of a data migration device according to an exemplary embodiment of the present disclosure, which is applied to a first node, such as a storage node, and can include: Figure 4 As shown in the figure, the data migration device can include:

[0156] The creating module 41 is configured to determine a regular storage area to be subjected to data migration in the first index disk, and create a first custom storage area and a second custom storage area associated with the regular storage area.

[0157] The first writing module 42 is configured to write historical object metadata into the first custom storage area, the historical object metadata being object metadata written in the regular storage area before starting data migration.

[0158] The first migration module 43 is configured to, in response to a data migration instruction, migrate the historical object metadata in the first customized storage area to a second index disk, and write process object metadata into the second customized storage area, the process object metadata being object metadata written into the regular storage area in the process of migrating the historical object metadata.

[0159] The second migration module 44 is configured to, in response to completion of migrating the historical object metadata, migrate the process object metadata in the second customized storage area to the second index disk.

[0160] The second index disk is managed by a second node, and the second node is configured to, in response to completion of migrating the historical object metadata, synchronously receive newly added object metadata from the first node and start a data migration verification operation of the second index disk.

[0161] Optionally, the first write module 42, when used for writing historical object metadata into the first customized storage area, includes:

[0162] a key of the historical object metadata written into the first customized storage area.

[0163] Correspondingly, the first migration module 43, when used for migrating the historical object metadata in the first customized storage area to a second index disk, includes:

[0164] obtaining a corresponding value from the regular storage area according to a key of the historical object metadata stored in the first customized storage area, to obtain complete historical object metadata.

[0165] migrating the historical object metadata to the second index disk.

[0166] and / or

[0167] The first write module 42, when used for writing process object metadata into the second customized storage area, includes:

[0168] a key of the process object metadata written into the second customized storage area.

[0169] Correspondingly, the first migration module 43, when used for migrating the process object metadata in the second customized storage area to the second index disk, includes:

[0170] obtaining a corresponding value from the regular storage area according to a key of the process object metadata stored in the second customized storage area, to obtain complete process object metadata.

[0171] migrating the process object metadata to the second index disk.

[0172] Optionally, the first migration module 43, when used for migrating the historical object metadata in the first custom storage area to the second index disk, comprises:

[0173] sequentially scanning each historical object metadata in the first custom storage area, in response to the number of scanned historical object metadata reaching a preset first threshold, constructing a data migration request containing the scanned historical object metadata of the preset first threshold number, and initiating the data migration request to the second node, so that the second node writes the historical object metadata contained in each data migration request received to the second index disk.

[0174] and / or

[0175] The first migration module 43, when used for migrating the process object metadata in the second custom storage area to the second index disk, comprises:

[0176] sequentially scanning each process object metadata in the second custom storage area, in response to the number of scanned process object metadata reaching a preset first threshold, constructing a data migration request containing the scanned process object metadata of the preset first threshold number, and initiating the data migration request to the second node, so that the second node writes the process object metadata contained in each data migration request received to the second index disk.

[0177] Optionally, the first migration module 43, when used for initiating the data migration request to the second node, comprises:

[0178] sending the data migration request to a data migration management module, so that the data migration management module, upon receiving the data migration request, if it is determined that the number of data migration requests in a sending queue does not reach a preset second threshold, adds the data migration request to the sending queue and notifies the first node to continue scanning object metadata, so that the data migration management module sequentially sends each data migration request in the sending queue to the second node.

[0179] Optionally, if the data migration management module determines that the number of data migration requests in the sending queue has reached the preset second threshold, it caches the data migration request, and in the case that the number of data migration requests in the sending queue is less than a preset third threshold, adds the cached data migration request to the sending queue; the preset third threshold is less than or equal to the preset second threshold.

[0180] Figure 5 is a structural schematic diagram of another data migration device according to an exemplary embodiment of the present disclosure, which is applied to a second node, such as Figure 5 As shown, the data migration device can include:

[0181] A second writing module 51 is configured to write the received historical object metadata into the second index disk to complete migration of the historical object metadata.

[0182] A receiving module 52 is configured to, in response to completion of migration of the historical object metadata, synchronously receive newly added object metadata from the first node and start a data migration verification operation of the second index disk.

[0183] A third writing module 53 is configured to write the newly added object metadata into the second index disk and write the received process object metadata into the second index disk based on the data migration verification operation.

[0184] Optionally, the writing module 51, when used to write the received historical object metadata into the second index disk, includes:

[0185] writing the received historical object metadata into an SST file and inserting the SST file containing the historical object metadata into the second index disk through an SST file insertion interface of the second index disk;

[0186] or

[0187] The receiving module 52, when used to synchronously receive newly added object metadata, includes:

[0188] writing the received newly added object metadata into an SST file and inserting the SST file containing the newly added object metadata into the second index disk through an SST file insertion interface of the second index disk;

[0189] or

[0190] The third writing module 53, when used to write the received process object metadata into the second index disk, includes:

[0191] writing the received process object metadata into an SST file and inserting the SST file containing the process object metadata into the second index disk through an SST file insertion interface of the second index disk.

[0192] Figure 6 is a structural schematic diagram of another data migration device according to an exemplary embodiment of the present disclosure, which is applied to a service node, such asFigure 6 The data migration apparatus can include, as shown:

[0193] A determination module 61 is configured to determine a regular storage area to be migrated in a first index disk of an object storage, and create a first custom storage area and a second custom storage area associated with the regular storage area.

[0194] A fourth writing module 62 is configured to write historical object metadata in the first custom storage area, the historical object metadata being object metadata written in the regular storage area before data migration starts.

[0195] A third migration module 63 is configured to, in response to a data migration instruction, migrate the historical object metadata in the first custom storage area to a second index disk, and write procedural object metadata in the second custom storage area, the procedural object metadata being object metadata written in the regular storage area during migration of the historical object metadata.

[0196] A fifth writing module 64 is configured to, in response to completion of migration of the historical object metadata, synchronously write new object metadata to the regular storage area and the second index disk, and migrate the procedural object metadata in the second custom storage area to the second index disk after starting a data migration verification operation of the second index disk.

[0197] For the apparatus embodiment, since it basically corresponds to the method embodiment, the relevant part can be referred to the part of the method embodiment. The apparatus embodiment described above is merely illustrative, wherein the units described above as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present disclosure according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0198] Correspondingly, the present disclosure provides an electronic device. It includes a processor, a memory for storing processor-executable instructions, wherein the processor is configured to execute the data migration method of any embodiment of the present disclosure.

[0199] In an exemplary embodiment, a non-transitory computer-readable storage medium, such as a memory including instructions, is also provided, which, when executed by a processor of an electronic device, enables the electronic device 600 to perform the data migration method of any embodiment of the present disclosure.

[0200] The non-transitory computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0201] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the present disclosure. It is intended that the present disclosure cover any and all variations of the present disclosure including those variations that can be incorporated into other forms, methods and apparatuses of the present technology without departing from the spirit or essential characteristics thereof. It is intended that the present disclosure include all those elements which are the case of equivalents of the elements disclosed above and that the elements substituted for those disclosed do not depart from the spirit of the present disclosure or essential characteristic thereof. The specification and examples are to be regarded as exemplary in nature and the true scope and spirit of the present disclosure is indicated by the following claims.

[0202] It is to be understood that the present disclosure is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A data migration method, characterized by, The method is applied to a first node that manages a first index disk of object storage, and the method comprises: determining a regular storage area to be migrated in the first index disk, and creating a first custom storage area and a second custom storage area associated with the regular storage area; writing historical object metadata into the first custom storage area, the historical object metadata being object metadata written into the regular storage area before data migration starts; in response to a data migration instruction, migrating the historical object metadata in the first custom storage area to a second index disk, and writing process object metadata into the second custom storage area, the process object metadata being object metadata written into the regular storage area in the process of migrating the historical object metadata; in response to completing the migration of the historical object metadata, migrating the process object metadata in the second custom storage area to the second index disk; wherein the second index disk is managed by a second node, and the second node synchronously receives new object metadata from the first node and starts a data migration verification operation of the second index disk in response to completing the migration of the historical object metadata.

2. The method of claim 1, wherein: the writing of the historical object metadata into the first custom storage area comprises: writing keys of the historical object metadata into the first custom storage area; correspondingly, the migrating of the historical object metadata in the first custom storage area to the second index disk comprises: obtaining corresponding values of the historical object metadata from the regular storage area according to the keys of the historical object metadata stored in the first custom storage area, to obtain complete historical object metadata; migrating the historical object metadata to the second index disk; and / or the writing of the process object metadata into the second custom storage area comprises: writing keys of the process object metadata into the second custom storage area; correspondingly, the migrating of the process object metadata in the second custom storage area to the second index disk comprises: obtaining corresponding values of the process object metadata from the regular storage area according to the keys of the process object metadata stored in the second custom storage area, to obtain complete process object metadata; migrating the process object metadata to the second index disk.

3. The method of claim 1, wherein: the migrating of the historical object metadata in the first custom storage area to the second index disk comprises: sequentially scanning each historical object metadata in the first custom storage area, and in response to the number of scanned historical object metadata reaching a preset first threshold, constructing a data migration request containing the preset first threshold number of scanned historical object metadata, and initiating the data migration request to the second node, so that the second node writes the historical object metadata contained in each received data migration request to the second index disk in response to the received data migration request. ​ and / or The migrating the process object metadata in the second custom storage area to the second index disk comprises: sequentially scanning each process object metadata in the second custom storage area, in response to the number of scanned process object metadata reaching a preset first threshold, constructing a data migration request containing the preset first threshold number of scanned process object metadata, and initiating the data migration request to the second node, so that the second node writes the process object metadata contained in each data migration request received to the second index disk in response to the data migration request.

4. The method of claim 3, wherein, The initiating the data migration request to the second node comprises: sending the data migration request to a data migration management module, so that the data migration management module, upon receiving the data migration request, adds the data migration request to a sending queue and notifies the first node to continue scanning object metadata if it is determined that the number of data migration requests in the sending queue has not reached a preset second threshold, and in turn causes the data migration management module to sequentially send each data migration request in the sending queue to the second node.

5. The method of claim 4, wherein, Wherein, If the data migration management module determines that the number of data migration requests in the sending queue has reached the preset second threshold, it caches the data migration request, and in the case that the number of data migration requests in the sending queue is less than a preset third threshold, it adds the cached data migration request to the sending queue; The preset third threshold is less than or equal to the preset second threshold.

6. A data migration method, characterized by, Wherein, The first node managing the first index disk of the object storage performs: determining a regular storage area to be data migrated in the first index disk, and creating a first custom storage area and a second custom storage area associated with the regular storage area; writing historical object metadata into the first custom storage area, the historical object metadata being object metadata written into the regular storage area before data migration starts; in response to a data migration instruction, migrating the historical object metadata in the first custom storage area to a second index disk, and writing process object metadata into the second custom storage area, the process object metadata being object metadata written into the regular storage area during the migration of the historical object metadata; in response to completing the migration of the historical object metadata, migrating the process object metadata in the second custom storage area to the second index disk; The method is applied to a second node, and the second node manages the second index disk, and the method comprises: writing the received historical object metadata to the second index disk to complete the migration of the historical object metadata; in response to completing the migration of the historical object metadata, synchronously receiving newly added object metadata with the first node and starting a data migration verification operation of the second index disk; in response to completing the migration of the historical object metadata, synchronously receiving newly added object metadata with the first node and starting a data migration verification operation of the second index disk; write the received incremental object metadata into the second index disk, and based on the data migration verification operation, write the received procedural object metadata into the second index disk.

7. The method of claim 6, wherein, the writing the received historical object metadata into the second index disk comprises: writing the received historical object metadata into an SST file, and inserting the SST file containing the historical object metadata into the second index disk through an SST file insertion interface of the second index disk; or the synchronously receiving incremental object metadata comprises: writing the received incremental object metadata into an SST file, and inserting the SST file containing the incremental object metadata into the second index disk through an SST file insertion interface of the second index disk; or the writing the received procedural object metadata into the second index disk comprises: writing the received procedural object metadata into an SST file, and inserting the SST file containing the procedural object metadata into the second index disk through an SST file insertion interface of the second index disk.

8. A data migration method, characterized by, The method comprises: determining a regular storage area to be migrated in a first index disk of object storage, and creating a first custom storage area and a second custom storage area associated with the regular storage area; writing historical object metadata into the first custom storage area, the historical object metadata being object metadata written into the regular storage area before data migration starts; in response to a data migration instruction, migrating the historical object metadata in the first custom storage area to a second index disk, and writing procedural object metadata into the second custom storage area, the procedural object metadata being object metadata written into the regular storage area during migration of the historical object metadata; in response to completion of migration of the historical object metadata, synchronously writing incremental object metadata into the regular storage area and the second index disk, and after starting a data migration verification operation of the second index disk, migrating the procedural object metadata in the second custom storage area to the second index disk.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by a processor to implement the steps of the method of any one of claims 1-8.

10. An electronic device, comprising: comprises: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the steps of the method of any one of claims 1-8.

Images