Data migration method and device of storage pool, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data migration method and device of a storage pool, electronic equipment and a storage medium, the storage pool comprises a metadata server and an object storage server, the metadata server is used for storing metadata of an object, the metadata comprises a storage position of data of the object, and the object storage server is used for storing the data of the object. And deleting a version needing to be deleted in the metadata of the target object in the first metadata server, determining the version needing to be deleted through deletion representation existing in the metadata of the target object in the second metadata server, and migrating data corresponding to the metadata of the remaining target object to a second object storage server. And the storage positions in the metadata of the remaining target objects are updated and then written into the second metadata server in batches, so that the total data of the first storage pool is migrated to the second storage pool, the problem of data consistency is avoided, and the user experience is improved in the total data migration process.

Description

Technical Field

[0001] The present invention relates to the technical field of data migration, and in particular to a method, device, electronic device and storage medium for data migration of a storage pool. Background Art

[0002] When dealing with massive data storage, multiple storage pools are usually built, and new storage pools are flexibly expanded according to the growth of data volume. Due to differences in the construction time, device configuration and service life of different storage pools, the storage pools show significant imbalance in overall capacity, remaining available capacity, device failure rate and data read and write performance. Specifically, the devices in the old storage pool have a relatively high failure rate due to long-term operation, and the data read and write performance may also decline. In order to ensure the stability and efficiency of data storage, these old storage pools must be updated or offline as necessary. For the update of some old devices, data reconstruction is required within the storage pool to ensure the integrity and continuity of the data during the update process. For the offline of the entire old storage pool, all data in the pool needs to be safely and efficiently migrated to other storage pools to avoid data loss or service interruption.

[0003] Since the data volume of a single storage pool is as high as the PB level, full migration of the entire storage pool will face many challenges, such as data consistency during the migration process, which will affect the user experience during the data migration process. Summary of the invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a method, device, electronic device and storage medium for data migration of a storage pool that overcomes the above problems or at least partially solves the above problems.

[0005] In order to solve the above problem, an embodiment of the present invention discloses a method for data migration of a storage pool, wherein the storage pool includes a metadata server and an object storage server; the metadata server is used to store metadata of an object, an object has at least one version of metadata, the version number of the version is identified by the timestamp of the creation time of the version metadata, and the metadata includes the storage location of the data of the object; the object storage server is used to store the data of the object; the method includes:

[0006] Querying metadata of the target object from a first metadata server in the first storage pool and a second metadata server in the second storage pool respectively to obtain metadata of the first at least one version and metadata of the second at least one version;

[0007] Determining that a version with a deletion mark exists in the metadata of the second at least one version;

[0008] Eliminate metadata of a version having a deletion mark in the metadata of the second at least one version from the metadata of the first at least one version, to obtain candidate metadata of at least one version;

[0009] Migrating data of each version from a first object storage server in the first storage pool to a second object storage server in the second storage pool according to the candidate metadata of the at least one version, and obtaining a target storage location of the data of each version;

[0010] Replacing the storage locations of the data of each version in the at least one version of the candidate metadata with the target storage locations of the data of each version, thereby obtaining the target metadata of the at least one version;

[0011] The target metadata of the at least one version is written in batches to the second metadata server.

[0012] Optionally, the method further includes:

[0013] In the case that the metadata of the second at least one version does not exist, the target metadata of the at least one version is written in batches to the second metadata server through atomic operations.

[0014] Optionally, migrating data of each version from the first object storage server to the second object storage server according to the candidate metadata of the at least one version includes:

[0015] Obtaining storage locations of data of each version according to the candidate metadata of the at least one version;

[0016] Migrate each version of the data from the first object storage server to the second object storage server according to the storage location of each version of the data.

[0017] Optionally, the first storage pool further includes a first interface server, and the method further includes:

[0018] When receiving a write request for the target object, querying metadata of the target object from the second metadata server through the first interface server;

[0019] If the metadata of the target object exists in the second metadata server, writing the metadata of the current version of the target object into the second metadata server;

[0020] If the metadata of the target object does not exist in the second metadata server, querying the metadata of the target object from the first metadata server;

[0021] If the metadata of the target object does not exist in the first metadata server, writing the metadata of the current version of the target object to the second metadata server;

[0022] If the metadata of the target object exists in the first metadata server, the metadata of the current version of the target object and the metadata of the target object existing in the first metadata server are written in batches to the second metadata server through atomic operations.

[0023] Optionally, the method further includes:

[0024] When receiving a deletion request for the target version of the target object, querying the metadata of the target object from the second metadata server through the first interface server;

[0025] If the metadata of the target version of the target object exists in the second metadata server, adding a deletion mark to the metadata of the target version of the target object;

[0026] If the metadata of the target object does not exist in the second metadata server, querying the metadata of the target object from the first metadata server;

[0027] If the metadata of the target version of the target object exists in the first metadata server, removing the metadata of the target version of the target object from the metadata of the target object in the first metadata server to obtain the metadata after removal;

[0028] If the removed metadata is not empty, writing the removed metadata to the second metadata server through an atomic operation;

[0029] If the removed metadata is empty, creating a first version of metadata of the target object according to the target version of metadata of the target object, wherein the first version of metadata contains a deletion mark;

[0030] The metadata of the first version is written to the second metadata server through an atomic operation.

[0031] Optionally, the method further includes:

[0032] When receiving a read request for the target version of the target object, querying the metadata of the target version of the target object from the second metadata server through the first interface server;

[0033] If the metadata of the target version of the target object exists in the second metadata server, and there is no deletion mark in the metadata of the target version of the target object, reading the data of the target version of the target object according to the storage location in the metadata of the target version of the target object;

[0034] If the metadata of the target version of the target object does not exist in the second metadata server, or a deletion mark exists in the metadata of the target version of the target object, querying the metadata of the target version of the target object from the first metadata server through the first interface server;

[0035] If the metadata of the target version of the target object exists in the first metadata server, data of the target version of the target object is read from the first object storage server according to a storage location in the metadata of the target version of the target object.

[0036] Optionally, the method further includes:

[0037] After all data of all objects in the first storage pool are migrated, traverse the second metadata server to determine the version of metadata with a deletion mark;

[0038] Delete the version of metadata that has a deletion marker.

[0039] Accordingly, an embodiment of the present invention discloses a device for data migration of a storage pool, wherein the storage pool includes a metadata server and an object storage server; the metadata server is used to store metadata of an object, an object has at least one version of metadata, the version number of the version is identified by a timestamp of the creation time of the version metadata, and the metadata includes a storage location of the data of the object; the object storage server is used to store the data of the object; the device includes:

[0040] A metadata query module, used to query metadata of a target object from a first metadata server in a first storage pool and a second metadata server in a second storage pool respectively, to obtain metadata of at least one first version and metadata of at least one second version;

[0041] A deletion mark version determination module, used to determine a version having a deletion mark in the metadata of the second at least one version;

[0042] a removal module, configured to remove metadata of a version having a deletion mark in the metadata of the second at least one version from the metadata of the first at least one version, so as to obtain candidate metadata of at least one version;

[0043] a migration module, configured to migrate data of each version from a first object storage server of the first storage pool to a second object storage server of the second storage pool according to the candidate metadata of the at least one version, and obtain a target storage location of the data of each version;

[0044] a target metadata determination module, configured to replace the storage locations of the data of each version in the candidate metadata of the at least one version with the target storage locations of the data of each version, so as to obtain the target metadata of the at least one version;

[0045] The target metadata writing module is used to write the target metadata of the at least one version to the second metadata server in batches.

[0046] Optionally, the device further comprises:

[0047] The first atomic operation batch writing module is used to write the target metadata of the at least one version to the second metadata server in batches through atomic operations when the metadata of the at least one second version does not exist.

[0048] Optionally, the migration module includes:

[0049] A storage location determination submodule, configured to obtain a storage location of data of each version according to the candidate metadata of the at least one version;

[0050] The migration submodule is used to migrate the data of each version from the first object storage server to the second object storage server according to the storage location of the data of each version.

[0051] Optionally, the first storage pool further includes a first interface server, and the device further includes:

[0052] A first query module, configured to query metadata of the target object from the second metadata server through the first interface server when receiving a write request for the target object;

[0053] A first writing module, configured to write the metadata of the current version of the target object into the second metadata server if the metadata of the target object exists in the second metadata server;

[0054] A second query module, configured to query the metadata of the target object from the first metadata server if the metadata of the target object does not exist in the second metadata server;

[0055] A second writing module, configured to write the metadata of the current version of the target object into the second metadata server if the metadata of the target object does not exist in the first metadata server;

[0056] The second atomic operation batch writing module is used to batch write the metadata of the current version of the target object and the metadata of the target object existing in the first metadata server into the second metadata server through atomic operations if the metadata of the target object exists in the first metadata server.

[0057] Optionally, the device further comprises:

[0058] A third query module, configured to query the metadata of the target object from the second metadata server through the first interface server upon receiving a deletion request for the target version of the target object;

[0059] a deletion mark adding module, configured to add a deletion mark to the metadata of the target version of the target object if the metadata of the target version of the target object exists in the second metadata server;

[0060] a fourth query module, configured to query the metadata of the target object from the first metadata server if the metadata of the target object does not exist in the second metadata server;

[0061] a module for determining metadata after removal, configured to remove the metadata of the target version of the target object from the metadata of the target object in the first metadata server if the metadata of the target version of the target object exists in the first metadata server, so as to obtain the metadata after removal;

[0062] a third atomic operation batch writing module, configured to write the removed metadata into the second metadata server through an atomic operation if the removed metadata is not empty;

[0063] A metadata creation module, configured to create a first version of metadata of the target object according to the metadata of the target version of the target object if the removed metadata is empty, wherein the metadata of the first version contains a deletion mark;

[0064] The fourth atomic operation batch writing module is used to write the metadata of the first version to the second metadata server through atomic operations.

[0065] Optionally, the device further comprises:

[0066] a fifth query module, configured to query the metadata of the target version of the target object from the second metadata server through the first interface server when receiving a read request for the target version of the target object;

[0067] A first reading module is used to read the target version of the target object data according to the storage location in the target version of the metadata of the target object if the metadata of the target version of the target object exists in the second metadata server and there is no deletion mark in the metadata of the target version of the target object;

[0068] a sixth query module, configured to query the metadata of the target version of the target object from the first metadata server through the first interface server if the metadata of the target version of the target object does not exist in the second metadata server, or a deletion mark exists in the metadata of the target version of the target object;

[0069] The second reading module is used to read the target version data of the target object from the first object storage server according to the storage position in the target version metadata of the target object if the target version metadata of the target object exists in the first metadata server.

[0070] Optionally, the device further comprises:

[0071] A traversal module, configured to traverse the second metadata server to determine a version of metadata with a deletion mark after all data of all objects in the first storage pool have been migrated;

[0072] The marked version deletion module is used to delete the version of metadata with a deletion mark.

[0073] Correspondingly, an embodiment of the present invention discloses an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the various steps of the above-mentioned storage pool data migration method embodiment when executed by the processor.

[0074] Accordingly, an embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each step of the above storage pool data migration method embodiment is implemented.

[0075] The embodiments of the present invention include the following advantages:

[0076] According to an embodiment of the present invention, a method for data migration of a storage pool is provided. The storage pool includes a metadata server and an object storage server. The metadata server is used to store metadata of an object. An object has at least one version of metadata. The version number of the version is identified by a timestamp of the creation time of the version metadata. The metadata includes a storage location of the data of the object. The object storage server is used to store the data of the object. The metadata of the target object is first queried from a first metadata server of a first storage pool and a second metadata server of a second storage pool respectively to obtain metadata of the first at least one version and metadata of the second at least one version. Then, it is determined that there is a version with a deletion mark in the metadata of the second at least one version. Then, from the metadata of the first at least one version, the metadata of the version with the deletion mark in the metadata of the second at least one version is removed to obtain candidate metadata of at least one version. Then, according to the candidate metadata of at least one version, each version of data is migrated from the first object storage server of the first storage pool to the second object storage server of the second storage pool, and a target storage location of each version of data is obtained. The storage location of each version of data in the candidate metadata of at least one version is replaced with the target storage location of each version of data to obtain target metadata of at least one version. The target metadata of at least one version is written to the second metadata server in batches. The embodiment of the present invention deletes the version to be deleted in the metadata of the target object in the first metadata server, determines the version to be deleted through the deletion representation in the metadata of the target object in the second metadata server, then migrates the data corresponding to the metadata of the remaining target objects to the second object storage server, and updates the storage location in the metadata of the remaining target objects and writes them in batches to the second metadata server, thereby realizing the migration of the full data of the first storage pool to the second storage pool, avoiding the problem of data consistency, and improving the user experience during the full data migration process. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 is a flowchart of the steps of a storage pool data migration method according to an embodiment of the present invention;

[0078] Figure 2 It is a scenario diagram of data migration of a storage pool according to an embodiment of the present invention;

[0079] Figure 3 It is a structural block diagram of a data migration device for a storage pool according to an embodiment of the present invention. DETAILED DESCRIPTION

[0080] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0081] Since the data volume of a single storage pool is as high as the PB level, full migration of the entire storage pool will face many challenges, such as data consistency during the migration process, which will affect the user experience during the data migration process.

[0082] One of the core concepts of an embodiment of the present invention is that the embodiment of the present invention deletes the version to be deleted in the metadata of the target object in the first metadata server, determines the version to be deleted through the deletion representation in the metadata of the target object in the second metadata server, and then migrates the data corresponding to the metadata of the remaining target object to the second object storage server, and updates the storage location in the metadata of the remaining target object and writes it in batches to the second metadata server, thereby realizing the migration of all data in the first storage pool to the second storage pool, avoiding the problem of data consistency, and improving user experience during the full data migration process.

[0083] Reference Figure 1 , shows a flowchart of the steps of a data migration method of a storage pool according to an embodiment of the present invention, wherein the storage pool includes a metadata server and an object storage server; the metadata server is used to store metadata of an object, an object has at least one version of metadata, the version number of the version is identified by the timestamp of the creation time of the version metadata, and the metadata includes the storage location of the data of the object; the object storage server is used to store the data of the object, and specifically may include the following steps:

[0084] Step 101 : query metadata of a target object from a first metadata server in a first storage pool and a second metadata server in a second storage pool respectively to obtain metadata of at least one first version and metadata of at least one second version.

[0085] Specifically, the first storage pool may be a source storage pool, the second storage pool may be a destination storage pool, and all data in the first storage pool needs to be migrated to the second storage pool. The metadata server is used to store metadata of objects, exemplarily in the form of key-value pairs.<k,v> The metadata of the object is stored. k is the identifier of the object, and v is the content of the metadata. Different versions of an object have the same k, but different versions have different v. The version number of each version is identified by the timestamp (ts) when the version was created, that is,<k,ts,v> For example, the first metadata server has metadata for object a, and a has three versions:<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3> , the metadata of object a also exists in the second metadata server, and a has three versions<a,t1,v1> ,<a,t2,v2^> ,<a,t3,v3> . Query the metadata of the target object a from the first metadata server of the first storage pool and the second metadata server of the second storage pool respectively, and obtain the metadata of at least one first version, that is, all versions of a on the first metadata server {<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3>}, and the metadata of at least one second version, i.e., all versions of a on the second metadata server {<a,t1,v1> ,<a,t2,v2^> ,<a,t3,v3>}.

[0086] Step 102: Determine whether a version with a deletion mark exists in the metadata of the second at least one version.

[0087] Specifically, it is determined that there is a version with a deletion mark in the metadata of the second at least one version. For example, the ^ symbol of v2^ is a deletion mark, that is,<a,t2,v2^> It is a version with a deletion mark, that is, version number t2 is a version with a deletion mark in the metadata of the second at least one version.

[0088] Step 103: Eliminate metadata of a version with a deletion mark in the metadata of the second at least one version from the metadata of the first at least one version, and obtain candidate metadata of at least one version.

[0089] Specifically, the metadata of the first at least one version is {<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3>}, version number t2 is the version with a deletion mark in the metadata of the second at least one version, and after removing the metadata of version number t2 from the metadata of the first at least one version, {<a,t1,v1> ,<a,t3,v3>}, that is, candidate metadata for at least one version.

[0090] Step 104: Migrate each version of data from the first object storage server of the first storage pool to the second object storage server of the second storage pool according to the candidate metadata of the at least one version, and obtain a target storage location of each version of data.

[0091] Specifically, according to at least one version of candidate metadata {<a,t1,v1> ,<a,t3,v3>}, migrate the data of version t1 and version t3 of the target object a from the first object storage server of the first storage pool to the second object storage server of the second storage pool. After migrating to the second object storage server of the second storage pool, obtain the data of version t1 and version t3 at the target storage location of the second object storage server respectively.

[0092] Step 105: Replace the storage location of each version of the data in the at least one version of candidate metadata with the target storage location of each version of the data to obtain the target metadata of at least one version.

[0093] Specifically, at least one version of the candidate metadata {<a,t1,v1> ,<a,t3,v3>}, the storage location in v1 is replaced by the target storage location of the t1 version of the data in the second object storage server, and the storage location in v3 is replaced by the target storage location of the t3 version of the data in the second object storage server, obtaining at least one version of the target metadata {<a,t1,v1'> ,<a,t3,v3'>}.

[0094] Step 106: write the at least one version of target metadata to the second metadata server in batches.

[0095] Specifically, get at least one version of the target metadata {<a,t1,v1'> ,<a,t3,v3'>},Will<a,t1,v1'> and<a,t3,v3'> Batch write to the second metadata server.

[0096] In an embodiment of the present invention, the method further includes:

[0097] In the case that the metadata of the second at least one version does not exist, the target metadata of the at least one version is written in batches to the second metadata server through atomic operations.

[0098] Specifically, in the case where the metadata of the second at least one version does not exist, illustratively, the metadata of object a exists in the first metadata server, and a has three versions:<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3> , the metadata of object a does not exist in the second metadata server, and at least one version of candidate metadata is {<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3>}, after migrating the data of version t1, version t2 and version t3 from the first object storage server to the second object storage server, the data of version t1, version t2 and version t3 are respectively obtained at the target storage location of the second object storage server, the storage location in v1 is replaced with the data of version t1 at the target storage location of the second object storage server, the storage location in v2 is replaced with the data of version t2 at the target storage location of the second object storage server, and the storage location in v3 is replaced with the data of version t3 at the target storage location of the second object storage server, and at least one version of target metadata is obtained {<a,t1,v1'> ,<a,t2,v2'> ,<a,t3,v3'>},Will{<a,t1,v1'> ,<a,t2,v2'> ,<a,t3,v3'>}Batch write to the second metadata server through atomic operations. If the write fails, repeat the above processing flow from the step of querying the metadata of the target object from the first metadata server and the second metadata server until success.

[0099] By writing to the second metadata server in batches through atomic operations, it is possible to ensure that the metadata update operation is atomic during the data migration process, thereby ensuring data consistency and integrity. Atomic operations mean that these operations are either all executed successfully or not executed at all, and there will be no situation where some operations succeed while others fail.

[0100] In the embodiment of the present invention, step 104 may specifically include the following sub-steps:

[0101] Sub-step S11, obtaining the storage location of each version of data according to the candidate metadata of the at least one version.

[0102] Specifically, following the above example, according to at least one version of candidate metadata {<a,t1,v1> ,<a,t3,v3>}, obtain the storage location of version t1 data in the first object storage server, and the storage location of version t3 data in the first object storage server.

[0103] Sub-step S12: Migrating each version of the data from the first object storage server to the second object storage server according to the storage location of each version of the data.

[0104] Specifically, based on the storage location of version t1 data in the first object storage server, version t1 data is migrated from the first object storage server to the second object storage server; based on the storage location of version t3 data in the first object storage server, version t3 data is migrated from the first object storage server to the second object storage server.

[0105] In the embodiment of the present invention, the first storage pool further includes a first interface server, and the method further includes:

[0106] When a write request for the target object is received, metadata of the target object is queried from the second metadata server through the first interface server.

[0107] Specifically, read and write redirection is performed first. Before data is ready to be migrated, that is, before time T0, the first interface server only retrieves the metadata of the object from the first metadata server, and reads, writes, and deletes the data of the object from the first object storage server; when data is ready to be migrated, that is, from time T0, the objects in the first storage pool are read only and not written, and all the data of the newly written objects are written to the second storage pool; refer to Figure 2 , shows a scenario diagram of data migration of a storage pool according to an embodiment of the present invention, where data migration begins from time T1 through a migration tool.

[0108] Starting from time T0, the user can initiate a write request for the target object through the client. When receiving the write request for the target object, the first interface server queries the metadata of the target object from the second metadata server. For example, the first metadata server contains metadata of object a and object b, where object a has three versions:<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3> , b has only one version<b,t0,v0> , the second metadata server does not have these two objects. After receiving the write request for the target object k, the first interface server queries the second metadata server for all versions of the metadata corresponding to the target object k.<k,ts,v> .

[0109] If the metadata of the target object exists in the second metadata server, the metadata of the current version of the target object is written to the second metadata server.

[0110] Specifically, if the metadata of the target object exists in the second metadata server, that is, the List is not empty and the metadata of other versions of the target object k exists in the List, then the data of the target object k in the write request of the target object k is written to the second object storage server, and the corresponding metadata is generated after writing.<k,t,v> , and then the metadata of the target object k<k,t,v> Write to the secondary metadata server.

[0111] If the metadata of the target object does not exist in the second metadata server, query the metadata of the target object from the first metadata server.

[0112] Specifically, if the metadata of the target object exists in the second metadata server, that is, List is empty, all versions List′ corresponding to the target object k are queried from the first metadata server.<k,ts,v> .

[0113] If the metadata of the target object does not exist in the first metadata server, the metadata of the current version of the target object is written to the second metadata server.

[0114] Specifically, if the metadata of the target object does not exist in the first metadata server, that is, List′ is empty, at this time, the metadata of the target object k does not exist in the first metadata server and the second metadata server, and the data of the target object k in the write request of the target object k is written to the second object storage server, and the corresponding metadata is generated after writing<k,t,v> , and then the metadata of the target object k<k,t,v> Write to the secondary metadata server.

[0115] If the metadata of the target object exists in the first metadata server, the metadata of the current version of the target object and the metadata of the target object existing in the first metadata server are written in batches to the second metadata server through atomic operations.

[0116] Specifically, if the metadata of the target object exists in the first metadata server, that is, List′ is not empty, at this time, the metadata of the target object k exists in the first metadata server, but not in the second metadata server, the metadata of the current version of the target object k is<k,t,v> Add to List' to get List", write List" in batches to the second metadata server through atomic operations. Continuing with the above example, if k == a is used as an example, retrieve the metadata List' corresponding to a from the first metadata server {<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3>}, the metadata of the current version of the target object k<k,t,v> Add to List′, get List″{<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3> ,<k,t,v>}, write List″ to the second metadata server in batches through atomic operations (if List is empty, write List″ in batches). If the write fails, retry from the step of querying the metadata of the target object in the second metadata server until success. For example, if List″{<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3> ,<k,t,v>}Failure in writing to the second metadata server as an atomic operation will result in<k,t,v> When writing to the second metadata server, it is possible that the atomic operation fails because two operations are performed on the metadata of the target object at the same time. If you try again, you will find that the metadata of the target object does not exist in the first metadata server. Therefore, you can directly write<k,t,v> Write to the second metadata server.

[0117] In an embodiment of the present invention, the method further includes:

[0118] When a deletion request for the target version of the target object is received, the metadata of the target object is queried from the second metadata server through the first interface server.

[0119] Specifically, upon receiving the target version for the target object k<k,ts> When a deletion request is received, the metadata list of all versions corresponding to k is queried from the second metadata server.<k,ts,v> .

[0120] If the metadata of the target version of the target object exists in the second metadata server, a deletion mark is added to the metadata of the target version of the target object.

[0121] Specifically, if List is not empty, but does not exist<k,ts> If the List is not empty, it will be returned directly.<k,ts> , then in the second metadata server<k,ts> A special column DELETE is added to the metadata v value to indicate the deletion mark, and v' is obtained to indicate that the version has been deleted, that is,<k,ts,v^> .

[0122] If the metadata of the target object does not exist in the second metadata server, query the metadata of the target object from the first metadata server.

[0123] Specifically, if List is empty, it is necessary to query all versions List′ corresponding to the target object k from the first metadata server.<k,ts,v> .

[0124] If the metadata of the target version of the target object exists in the first metadata server, the metadata of the target version of the target object is removed from the metadata of the target object in the first metadata server to obtain the removed metadata.

[0125] Specifically, if there is<k,ts> When<k,ts> Remove from List′, get List″<k,ts,v> , Continuing with the above example, taking k==a, t==t2 as an example, query the metadata List′{<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3>}, remove the metadata of version t2 to get List″{<a,t1,v1> ,<a,t3,v3>}.

[0126] If the removed metadata is not empty, the removed metadata is written to the second metadata server through an atomic operation.

[0127] Specifically, List″{<a,t1,v1> ,<a,t3,v3>}Write to the second metadata server through atomic operations (if List is empty, write to List″ in batches). If the write is successful, delete the data in the storage location of the metadata v2 of version t2 from the first object storage server. If the write fails, retry from the step of querying the metadata of the target object from the second metadata server until it succeeds. In fact, there may be a situation where the atomic operation fails to write because two operations are simultaneously performed on the metadata of the target object. In this case, retrying only requires adding a special column DELETE in v2 to obtain v2^.<a,t2,v2^> The data is written to the second metadata server, and then the corresponding data is deleted from the second object storage server.

[0128] If the removed metadata is empty, a first version of metadata of the target object is created according to the metadata of the target version of the target object, and a deletion mark exists in the metadata of the first version.

[0129] Specifically, if the metadata after removal is empty, that is, List″ is empty, that is, List′ only exists<k,ts> , before removing<k,ts> After that, List″ is empty. At this time, a special version needs to be constructed.<k,ts',v^> Added to List", this special version indicates that all versions of the object are deleted.

[0130] The metadata of the first version is written to the second metadata server through an atomic operation.

[0131] Specifically, List″ is written to the second metadata server through an atomic operation.

[0132] In an embodiment of the present invention, the method further includes:

[0133] When a read request for the target version of the target object is received, metadata of the target version of the target object is queried from the second metadata server through the first interface server.

[0134] Specifically, when a read request for a target version ts of a target object k is received, the first interface server first queries the second metadata server for the target version ts of the target object k.<k,ts> Corresponding metadata<k,ts,v> .

[0135] If the target version of the target object has metadata in the second metadata server and there is no deletion mark in the target version of the target object metadata, the target version of the target object data is read according to the storage location in the target version of the target object metadata.

[0136] Specifically, if there is<k,ts,v> , and v does not contain the special column DELETE, then the data is read from the first object storage server or the second object storage server according to the storage location recorded in v.

[0137] If the metadata of the target version of the target object does not exist in the second metadata server, or there is a deletion mark in the metadata of the target version of the target object, the metadata of the target version of the target object is queried from the first metadata server through the first interface server.

[0138] Specifically, if the second metadata server does not exist<k,ts,v> , then the first interface server obtains the first metadata server<k,ts> Corresponding metadata<k,ts,v'> .

[0139] If the metadata of the target version of the target object exists in the first metadata server, data of the target version of the target object is read from the first object storage server according to a storage location in the metadata of the target version of the target object.

[0140] Specifically, if there is<k,ts,v'> , then read the data from the first object storage server.

[0141] In an embodiment of the present invention, the method further includes:

[0142] After the data of all objects in the first storage pool are completely migrated, the second metadata server is traversed to determine the version of metadata with a deletion mark.

[0143] Specifically, traverse the first metadata server to obtain all versions of each object in turn.<k,ts,v> ,Through the above method, the data of each object is migrated through the migration tool, and at time T2, the migration of all object data is completed. The second storage pool also includes a second interface server. At time T3, all requests sent to the first interface server are redirected to the second interface server at the network layer. From time T3, the first interface server no longer receives any requests, and all requests are processed by the second interface server according to general logic. However, for read requests, if v contains the special column DELETE, it is considered that the version does not exist.

[0144] At time T3, the second metadata server is traversed by a cleaning tool to determine the version of metadata with a deletion mark.

[0145] Delete the version of metadata that has a deletion marker.

[0146] Specifically, for all versions of v that contain the special column DELETE<k,ts,v^> At T4, the deletion of all object versions containing the special column DELETE is completed. At T5, all requests are processed by various components in the second storage pool according to general logic.

[0147] The migration process of the embodiment of the present invention first redirects the read and write of the first metadata server, then migrates the data, and finally clears the special mark. Read and write redirection method: Before the migration is completed, the request for the first storage pool is still processed by the first metadata server. After the migration is completed, the request for the first storage pool is processed by the second metadata server. The method of ensuring data consistency during the migration process uses atomic operations and special columns to achieve metadata consistency processing. It has wide applicability and is suitable for full data migration between most object storage pools. It has low cost and does not affect the availability of services or the reliability of data during the migration process. It can truly complete the replacement of old storage pools without user perception, eliminating the risk of long-term operation of old equipment. In the process of migrating object data from the source storage pool to the destination storage pool, users can read, write and delete old objects from the source storage pool normally, and can also read, write and delete new objects from the destination resource pool. After the migration is completed, users can read, write and delete any object from the new storage pool without making any modifications.

[0148] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0149] Reference Figure 3 , shows a structural block diagram of a data migration device for a storage pool according to an embodiment of the present invention, wherein the storage pool includes a metadata server and an object storage server; the metadata server is used to store metadata of an object, an object has at least one version of metadata, the version number of the version is identified by a timestamp of the creation time of the version metadata, and the metadata includes a storage location of the data of the object; the object storage server is used to store the data of the object, and specifically may include the following modules:

[0150] The metadata query module 201 is used to query the metadata of the target object from the first metadata server of the first storage pool and the second metadata server of the second storage pool respectively, to obtain the metadata of the first at least one version and the metadata of the second at least one version;

[0151] A deletion mark version determining module 202, configured to determine a version having a deletion mark in the metadata of the second at least one version;

[0152] A removal module 203 is used to remove metadata of a version having a deletion mark in the metadata of the second at least one version from the metadata of the first at least one version, so as to obtain candidate metadata of at least one version;

[0153] A migration module 204, configured to migrate data of each version from a first object storage server of the first storage pool to a second object storage server of the second storage pool according to the candidate metadata of the at least one version, and obtain a target storage location of the data of each version;

[0154] The target metadata determination module 205 is used to replace the storage location of each version of the data in the at least one version of the candidate metadata with the target storage location of each version of the data, so as to obtain the target metadata of the at least one version;

[0155] The target metadata writing module 206 is configured to write the target metadata of the at least one version to the second metadata server in batches.

[0156] The device of the embodiment of the present invention deletes the version to be deleted in the metadata of the target object in the first metadata server, determines the version to be deleted through the deletion representation in the metadata of the target object in the second metadata server, and then migrates the data corresponding to the metadata of the remaining target objects to the second object storage server, and updates the storage location in the metadata of the remaining target objects and writes them in batches to the second metadata server, thereby realizing the migration of the full data of the first storage pool to the second storage pool, avoiding the problem of data consistency, and improving the user experience during the full data migration process.

[0157] In an embodiment of the present invention, the device further includes:

[0158] The first atomic operation batch writing module is used to write the target metadata of the at least one version to the second metadata server in batches through atomic operations when the metadata of the at least one second version does not exist.

[0159] In an embodiment of the present invention, the migration module includes:

[0160] A storage location determination submodule, configured to obtain a storage location of data of each version according to the candidate metadata of the at least one version;

[0161] The migration submodule is used to migrate the data of each version from the first object storage server to the second object storage server according to the storage location of the data of each version.

[0162] In the embodiment of the present invention, the first storage pool further includes a first interface server, and the device further includes:

[0163] A first query module, configured to query metadata of the target object from the second metadata server through the first interface server when receiving a write request for the target object;

[0164] A first writing module, configured to write the metadata of the current version of the target object into the second metadata server if the metadata of the target object exists in the second metadata server;

[0165] A second query module, configured to query the metadata of the target object from the first metadata server if the metadata of the target object does not exist in the second metadata server;

[0166] A second writing module, configured to write the metadata of the current version of the target object into the second metadata server if the metadata of the target object does not exist in the first metadata server;

[0167] The second atomic operation batch writing module is used to batch write the metadata of the current version of the target object and the metadata of the target object existing in the first metadata server into the second metadata server through atomic operations if the metadata of the target object exists in the first metadata server.

[0168] In an embodiment of the present invention, the device further includes:

[0169] A third query module, configured to query the metadata of the target object from the second metadata server through the first interface server upon receiving a deletion request for the target version of the target object;

[0170] a deletion mark adding module, configured to add a deletion mark to the metadata of the target version of the target object if the metadata of the target version of the target object exists in the second metadata server;

[0171] a fourth query module, configured to query the metadata of the target object from the first metadata server if the metadata of the target object does not exist in the second metadata server;

[0172] a module for determining metadata after removal, configured to remove the metadata of the target version of the target object from the metadata of the target object in the first metadata server if the metadata of the target version of the target object exists in the first metadata server, so as to obtain the metadata after removal;

[0173] a third atomic operation batch writing module, configured to write the removed metadata into the second metadata server through an atomic operation if the removed metadata is not empty;

[0174] A metadata creation module, configured to create a first version of metadata of the target object according to the metadata of the target version of the target object if the removed metadata is empty, wherein the metadata of the first version contains a deletion mark;

[0175] The fourth atomic operation batch writing module is used to write the metadata of the first version to the second metadata server through atomic operations.

[0176] In an embodiment of the present invention, the device further includes:

[0177] a fifth query module, configured to query the metadata of the target version of the target object from the second metadata server through the first interface server when receiving a read request for the target version of the target object;

[0178] A first reading module is used to read the target version of the target object data according to the storage location in the target version of the metadata of the target object if the metadata of the target version of the target object exists in the second metadata server and there is no deletion mark in the metadata of the target version of the target object;

[0179] a sixth query module, configured to query the metadata of the target version of the target object from the first metadata server through the first interface server if the metadata of the target version of the target object does not exist in the second metadata server, or a deletion mark exists in the metadata of the target version of the target object;

[0180] The second reading module is used to read the target version data of the target object from the first object storage server according to the storage position in the target version metadata of the target object if the target version metadata of the target object exists in the first metadata server.

[0181] In an embodiment of the present invention, the device further includes:

[0182] A traversal module, configured to traverse the second metadata server to determine a version of metadata with a deletion mark after all data of all objects in the first storage pool have been migrated;

[0183] The marked version deletion module is used to delete the version of metadata with a deletion mark.

[0184] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0185] An embodiment of the present invention further provides an electronic device, including:

[0186] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, each process of the method embodiment of data migration of the above-mentioned storage pool is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0187] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned method embodiment for migrating data from a storage pool can be implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0188] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0189] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0190] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0193] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0194] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0195] The above is a detailed introduction to the method, device, electronic device and storage medium for data migration of a storage pool provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for data migration of a storage pool, characterized in that: The storage pool includes a metadata server and an object storage server; the metadata server is used to store metadata of an object, an object has at least one version of metadata, the version number of the version is identified by the timestamp of the creation time of the version metadata, and the metadata includes the storage location of the data of the object; The object storage server is used to store the data of the object; the method comprises: Querying metadata of the target object from a first metadata server in the first storage pool and a second metadata server in the second storage pool respectively to obtain metadata of the first at least one version and metadata of the second at least one version; Determining that a version with a deletion mark exists in the metadata of the second at least one version; Eliminate metadata of a version having a deletion mark in the metadata of the second at least one version from the metadata of the first at least one version, to obtain candidate metadata of at least one version; Migrating data of each version from a first object storage server in the first storage pool to a second object storage server in the second storage pool according to the candidate metadata of the at least one version, and obtaining a target storage location of the data of each version; Replacing the storage location of each version of the data in the at least one version of the candidate metadata with the target storage location of each version of the data, thereby obtaining the target metadata of the at least one version; The target metadata of the at least one version is written in batches to the second metadata server.

2. The method according to claim 1, characterized in that The method further comprises: In the case that the metadata of the second at least one version does not exist, the target metadata of the at least one version is written in batches to the second metadata server through atomic operations.

3. The method according to claim 1, characterized in that: The step of migrating data of each version from the first object storage server to the second object storage server according to the candidate metadata of the at least one version includes: Obtaining storage locations of data of each version according to the candidate metadata of the at least one version; Migrate each version of the data from the first object storage server to the second object storage server according to the storage location of each version of the data.

4. The method according to claim 1, characterized in that The first storage pool further includes a first interface server, and the method further includes: When receiving a write request for the target object, querying metadata of the target object from the second metadata server through the first interface server; If the metadata of the target object exists in the second metadata server, writing the metadata of the current version of the target object into the second metadata server; If the metadata of the target object does not exist in the second metadata server, querying the metadata of the target object from the first metadata server; If the metadata of the target object does not exist in the first metadata server, writing the metadata of the current version of the target object to the second metadata server; If the metadata of the target object exists in the first metadata server, the metadata of the current version of the target object and the metadata of the target object existing in the first metadata server are written in batches to the second metadata server through atomic operations.

5. The method according to claim 4, characterized in that The method further comprises: When receiving a deletion request for the target version of the target object, querying the metadata of the target object from the second metadata server through the first interface server; If the metadata of the target version of the target object exists in the second metadata server, adding a deletion mark to the metadata of the target version of the target object; If the metadata of the target object does not exist in the second metadata server, querying the metadata of the target object from the first metadata server; If the metadata of the target version of the target object exists in the first metadata server, removing the metadata of the target version of the target object from the metadata of the target object in the first metadata server to obtain the metadata after removal; If the removed metadata is not empty, writing the removed metadata to the second metadata server through an atomic operation; If the removed metadata is empty, creating a first version of metadata of the target object according to the target version of metadata of the target object, wherein the first version of metadata contains a deletion mark; The metadata of the first version is written to the second metadata server through an atomic operation.

6. The method according to claim 4, characterized in that The method further comprises: When receiving a read request for the target version of the target object, querying the metadata of the target version of the target object from the second metadata server through the first interface server; If the metadata of the target version of the target object exists in the second metadata server, and there is no deletion mark in the metadata of the target version of the target object, reading the data of the target version of the target object according to the storage location in the metadata of the target version of the target object; If the metadata of the target version of the target object does not exist in the second metadata server, or a deletion mark exists in the metadata of the target version of the target object, querying the metadata of the target version of the target object from the first metadata server through the first interface server; If the metadata of the target version of the target object exists in the first metadata server, data of the target version of the target object is read from the first object storage server according to a storage location in the metadata of the target version of the target object.

7. The method according to claim 1, characterized in that The method further comprises: After all data of all objects in the first storage pool are migrated, traverse the second metadata server to determine the version of metadata with a deletion mark; Delete the version of metadata that has a deletion marker.

8. A device for data migration of a storage pool, characterized in that: The storage pool includes a metadata server and an object storage server; the metadata server is used to store metadata of an object, an object has at least one version of metadata, the version number of the version is identified by the timestamp of the creation time of the version metadata, and the metadata includes the storage location of the data of the object; The object storage server is used to store the data of the object; the device includes: A metadata query module, used to query metadata of a target object from a first metadata server in a first storage pool and a second metadata server in a second storage pool respectively, to obtain metadata of at least one first version and metadata of at least one second version; A deletion mark version determination module, used to determine a version having a deletion mark in the metadata of the second at least one version; a removal module, configured to remove metadata of a version having a deletion mark in the metadata of the second at least one version from the metadata of the first at least one version, so as to obtain candidate metadata of at least one version; a migration module, configured to migrate data of each version from a first object storage server of the first storage pool to a second object storage server of the second storage pool according to the candidate metadata of the at least one version, and obtain a target storage location of the data of each version; a target metadata determination module, configured to replace the storage locations of the data of each version in the candidate metadata of the at least one version with the target storage locations of the data of each version, so as to obtain the target metadata of the at least one version; The target metadata writing module is used to write the target metadata of the at least one version to the second metadata server in batches.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the method for data migration of a storage pool as claimed in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for data migration of a storage pool as claimed in any one of claims 1 to 7 are implemented.

Images