Data migration method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a data migration method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: in a data migration process, in response to a data reading request sent by a terminal, reading data based on a source resource pool and a target resource pool; in the data migration process, in response to a data writing request sent by a terminal, performing data writing based on the target resource pool; in a data migration process, in response to a data deletion request sent by a terminal, performing data deletion based on the target resource pool; wherein the data migration is used for migrating the object data and the metadata in the source resource pool into the target resource pool. By adopting the method, the data migration can be realized under the condition of not influencing the application programming interface to process the user request, and the strong consistency of the data in the migration process is ensured.

Description

Technical Field

[0001] The present application relates to the technical field of data migration, and in particular to a data migration method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] As time goes by and technology advances, old storage devices age and malfunction frequently, while new storage devices have excellent, stable and reliable performance. The need to migrate data from the old resource pool to the new resource pool is becoming increasingly urgent.

[0003] Ensuring data consistency is the core premise of data migration. On this basis, not affecting user business processing and achieving user-unawareness is the focus of data migration. However, in order to ensure data consistency, the existing data migration methods generally adopt offline migration solutions. During the migration process, it is impossible to ensure that the application programming interface (API) can read, write and delete data. This method seriously affects the user experience. Some data migration methods achieve final consistency by merging source data, target data, and modifying operations, but they cannot guarantee strong consistency of data during the migration process.

[0004] Therefore, how to achieve data migration without affecting the API's processing of user requests and ensure strong consistency of data during the migration process remains an urgent issue to be solved. Summary of the invention

[0005] Based on this, it is necessary to provide a data migration method, device, computer equipment, storage medium and program product that can realize data migration without affecting the API processing of user requests and ensure strong consistency of data during the migration process to address the above technical problems.

[0006] In a first aspect, the present application provides a data migration method, the method comprising:

[0007] During the data migration process, in response to a data read request sent by the terminal, data is read based on the source resource pool and the destination resource pool;

[0008] During the data migration process, in response to a data write request sent by the terminal, data is written based on the destination resource pool;

[0009] During the data migration process, data deletion is performed based on the destination resource pool in response to a data deletion request sent by the terminal; wherein the source resource pool only supports reading operations of object data and metadata, and the destination resource pool supports reading operations, writing operations, and deleting operations of object data and metadata, and the metadata is used to describe the object data; wherein data migration is used to migrate object data and metadata in the source resource pool to the destination resource pool.

[0010] In one embodiment, the data deletion request is used to request deletion of first object data and first metadata of a first object; and the data deletion based on the destination resource pool includes:

[0011] Writing a deletion mark column of the first metadata into the destination resource pool;

[0012] If the data columns of the first object data and the first metadata exist in the destination resource pool, deleting the first object data and the first metadata based on the deletion mark column of the first metadata;

[0013] If the data columns of the first object data and the first metadata do not exist in the destination resource pool, the first object data and the second metadata are not migrated during the data migration process.

[0014] In one embodiment, the data read request is used to request to read second object data and second metadata of the second object; and the data reading based on the source resource pool and the destination resource pool includes:

[0015] If the destination resource pool contains a deletion mark column for the second metadata, the output data reading result is that the second object does not exist;

[0016] If the destination resource pool does not contain a deletion mark column of the second metadata, and the destination resource pool or the source resource pool contains a data column of the second metadata, the second object data is read according to the data column of the second metadata, and the data reading result is output as the second object data and the second metadata;

[0017] If the data column of the second metadata does not exist in the destination resource pool and the source resource pool, the output data reading result is that the second object does not exist.

[0018] In one embodiment, the data write request is used to request writing third object data and third metadata of a third object; and the data writing based on the destination resource pool includes:

[0019] If the original object data and original metadata of the third object exist in the destination resource pool, the third object data is written into the destination resource pool and the original object data is overwritten, and the third metadata is written into the destination resource pool and the original metadata is overwritten.

[0020] In one embodiment, the source resource pool is constructed based on multiple servers; before the data migration, the method further includes:

[0021] Obtaining an object to be migrated specified by each server, wherein the object to be migrated includes object data to be migrated and metadata to be migrated;

[0022] For each server, create a migration thread for the object to be migrated in the thread pool;

[0023] Accordingly, the data migration process includes:

[0024] When there is no data column of the metadata to be migrated and no deletion mark column in the destination resource pool, the object data to be migrated and the metadata to be migrated in the original resource pool are migrated to the destination resource pool.

[0025] In one embodiment, the method further comprises:

[0026] After completing the data migration, for the object data to be deleted and the metadata to be deleted included in the object to be deleted, if there is a deletion mark column of the metadata to be deleted in the destination resource pool, and there is a data column of the metadata to be deleted in the destination resource pool, then delete the data column of the metadata to be deleted, the deletion mark column of the metadata to be deleted, and the object data to be deleted;

[0027] If the destination resource pool has a deletion marker column of the metadata to be deleted, and the resource pool does not have a data column of the metadata to be deleted, the deletion marker column of the metadata to be deleted is deleted.

[0028] In a second aspect, the present application provides a data migration device, the device comprising:

[0029] A data reading module is used to read data based on a source resource pool and a destination resource pool in response to a data reading request sent by a terminal during data migration;

[0030] A data writing module, used for responding to a data writing request sent by a terminal during data migration and writing data based on the destination resource pool;

[0031] A data deletion module is used to delete data based on the destination resource pool in response to a data deletion request sent by a terminal during data migration; wherein the source resource pool only supports reading operations of object data and metadata, and the destination resource pool supports reading operations, writing operations, and deleting operations of object data and metadata, and the metadata is used to describe the object data; wherein data migration is used to migrate object data and metadata in the source resource pool to the destination resource pool.

[0032] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect are implemented.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0034] According to a fifth aspect, a computer program product comprises a computer program, wherein when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0035] The above-mentioned data migration method, device, computer equipment, storage medium and program product. To ensure data consistency, before data migration, set the source resource pool to only support reading, not writing and deletion, to ensure that the source resource pool will not increase or decrease data. During the data migration process, if a user request sent by the terminal (such as a data read request, a data write request, and a data delete request) is received, it will not affect the API module to process the user request and respond to the user request in real time. When responding to user requests, data write requests and data delete requests are only performed in the destination resource pool, not in the source resource pool, so as to ensure the consistency of data migration. The method provided in this embodiment can realize data migration without affecting the API module to process user requests, and ensure the strong consistency of data during the migration process. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 An application environment diagram of a data migration method in an embodiment;

[0038] Figure 2 A schematic diagram of a data migration method in one embodiment;

[0039] Figure 3 A schematic diagram of a flow chart of an application programming interface in a data migration system in one embodiment;

[0040] Figure 4 A partial flow chart of a data migration method in one embodiment;

[0041] Figure 5A partial flow chart of a data migration method in another embodiment;

[0042] Figure 6 A partial flow chart of a data migration method in another embodiment;

[0043] Figure 7 A partial flow chart of a data migration method in another embodiment;

[0044] Figure 8 A partial flow chart of a data migration method in another embodiment;

[0045] Fig. 9 A partial flow chart of a data migration method in another embodiment;

[0046] Fig.10 is a schematic diagram of the structure of a data migration device in one embodiment;

[0047] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] The data migration method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0050] In an exemplary embodiment, Figure 2 As shown, a data migration method is provided, which is applied to Figure 1The server 104 in the example is used as an example to illustrate, including the following steps 201 to 203. Among them:

[0051] Step 201 : During data migration, in response to a data read request sent by a terminal, data is read based on a source resource pool and a destination resource pool.

[0052] Data migration is used to migrate object data and metadata from the source resource pool to the destination resource pool. The metadata is used to describe the object data. It provides information about the object but does not contain the content of the object itself, allowing the system to effectively manage, retrieve and protect the stored data. The resource pool is a logical concept used to manage and organize storage resources. It combines a group of physical storage devices (such as hard disk drives, disk arrays, tape libraries, flash memory devices, etc.) into a collection to form a unified logical storage unit, thereby providing centralized management and flexible allocation of storage resources. The source resource pool can be understood as the old resource pool, and the destination resource pool can be understood as the new resource pool.

[0053] The source resource pool only supports the read operation of object data and metadata, and the destination resource pool supports the read operation, write operation, and delete operation of object data and metadata.

[0054] In one embodiment, the data migration method is implemented based on a migration system, and the migration system includes an application programming interface (Application Programming Interface, API for short) module and a migration module.

[0055] Before data migration begins, set the migration system's API module to handle user business requests. In addition, set the source resource pool to only support read operations on object data and metadata, and set the destination resource pool to support read operations, write operations, and delete operations on object data and metadata. See Figure 3 The API module shown in the figure implements the flow chart of source resource pool reading, destination resource pool reading, destination resource pool writing and destination resource pool deletion. Figure 3 As shown, the API module obtains object data and metadata of the object data through a terminal (Client). The source resource pool and the destination resource pool correspond to different terminals respectively.

[0056] During data migration, the migration module migrates the object data and metadata of the source resource pool to the destination resource pool. At the same time, the API module handles user business needs. After the data migration is completed, the API module is switched to the API module of the destination resource pool.

[0057] It should be noted that during the data migration process, in order not to affect the API module's processing of user requests and to achieve user-unawareness during data migration, if a data read request is received during the data migration process, it is necessary to respond to the data read request and reply. Specifically, the object data and metadata that the data read request wants to read are first queried in the destination resource pool. If the object data and metadata that the data read request wants to read are not in the destination resource pool, the object data and metadata that the data read request wants to read are queried in the source resource pool. After the object data and metadata that the data read request wants to read are queried, the object data and metadata are replied. If the object data and metadata that the data read request wants to read are not queried, the reply is that the data does not exist or the object does not exist.

[0058] In addition, it should be noted that only when the migration metadata is completed, the object is considered to be migrated. If only the object data is migrated, the API module cannot read, write or delete the object, and the object data is garbage data. Preferably, the migration module uses the Compare And Swap (CAS) method to migrate metadata to ensure that when the API module writes concurrently with the migration write, the metadata written by the API module will not be overwritten.

[0059] Step 202: During data migration, in response to a data write request sent by a terminal, data is written based on the destination resource pool.

[0060] During the data migration process, in order not to affect the API module's processing of user requests and to achieve user-unawareness during data migration, if a data write request is received during the data migration process, it is necessary to respond to the data write request and reply. Specifically, the object data and metadata requested by the data write request are written into the destination resource pool.

[0061] Step 203: During the data migration process, in response to a data deletion request sent by the terminal, data is deleted based on the destination resource pool.

[0062] During the data migration process, in order not to affect the API module's processing of user requests and to achieve data migration without user awareness, if a data deletion request is received during the data migration process, it is necessary to respond to the data write request and reply. Specifically, the object data and metadata requested to be deleted by the data deletion request are deleted in the destination resource pool.

[0063] In the above data migration method, in order to ensure data consistency, before data migration, the source resource pool is set to only support reading, not writing and deletion, to ensure that the source resource pool will not increase or decrease data. During the data migration process, if a user request sent by the terminal (such as a data read request, a data write request, and a data delete request) is received, it will not affect the API module to process the user request and respond to the user request in real time. When responding to user requests, data write requests and data delete requests are only performed in the destination resource pool, not in the source resource pool, so as to ensure the consistency of data migration. The method provided in this embodiment can realize data migration without affecting the API module to process user requests, and ensure the strong consistency of data during the migration process.

[0064] In an exemplary embodiment, the data deletion request is used to request deletion of first object data and first metadata of a first object. Figure 4 As shown, step 203 performs data deletion based on the destination resource pool, including steps 401 to 403, wherein:

[0065] Step 401: Write a deletion mark column of the first metadata into the destination resource pool.

[0066] The deletion mark column can be understood as a string of characters, which is used to identify the data column of the first metadata as the data column to be deleted, and the first object data corresponding to the first metadata as the object data to be deleted. The data column can be understood as a field included in the metadata.

[0067] Step 402: If the data columns of the first object data and the first metadata exist in the destination resource pool, the first object data and the first metadata are deleted based on the deletion mark column of the first metadata.

[0068] If the data columns of the first object data and the first metadata exist in the destination resource pool, the first object data and the first metadata are deleted.

[0069] Step 403: If the data columns of the first object data and the first metadata do not exist in the destination resource pool, the first object data and the second metadata are not migrated during the data migration process.

[0070] If the data columns of the first object data and the first metadata do not exist in the destination resource pool, it means that the first object data and the first metadata have not been migrated to the destination resource pool. At this time, since the source resource pool does not support the deletion operation, in order to simplify the deletion process, the first object data and the second metadata are not migrated during the data migration process.

[0071] In one embodiment, see Figure 5The flowchart of the API module processing data deletion request is shown. Since the source resource pool is set to not support data deletion operations during migration, when the API module processes the data deletion request, it first writes the deletion mark column of the first metadata in the destination resource pool. If the first object to be deleted is an object newly written into the destination resource pool during migration, since the data column of the first metadata and the first object data exist in the destination resource pool, the first metadata and the first object data can be directly deleted. If the first object to be deleted is an object in the source resource pool that has been migrated to the destination resource pool, since the data column of the first metadata and the first object data exist in the destination resource pool, the first metadata and the first object data can also be directly deleted. If the first object to be deleted is an object in the source resource pool that has not been migrated, a direct reply of deletion success is made. In the subsequent migration process, the migration module first queries the deletion mark column of the first metadata before migration. Because the first metadata deletion mark column exists, the metadata and data of the object are not migrated.

[0072] Specifically, the API module processes the data deletion request and executes step 501 to write the deletion mark column of the first metadata to the destination resource pool. If the destination resource pool does not have the data column of the first metadata, it indicates that the object to be deleted is an unmigrated source resource pool object. Since the destination resource pool does not have the metadata and data of the object, there is no need to perform the deletion operation, and the process proceeds to step 502. If the destination resource pool has the data column of metadata, it indicates that the object to be deleted is an object newly written into the destination resource pool during migration or an object in the source resource pool that has been migrated to the destination resource pool. At this time, the process proceeds to step 503 to delete the data column of the first metadata in the destination resource pool and delete the first object data.

[0073] In one embodiment, in the migration system, the API module first processes the data deletion request and then migrates the data. Specifically, the API module first writes the deletion mark column of the first metadata. At this time, the first object in the source resource pool has not been migrated, and the data column of the first metadata does not exist. The API module does not need to delete the data column of the first metadata and the first object data. Before performing data migration, the migration module first queries the destination resource pool and finds that there is a deletion mark column for the first metadata. Therefore, there is no need to migrate the first metadata and the first object data in the source resource pool. The migration system also includes a cleanup module. After the data migration is completed, there is a deletion mark column for the first metadata in the destination resource pool for the first object, and the deletion mark column for the first metadata is deleted by the cleanup module.

[0074] In one embodiment, in a migration system, the migration module migrates first, and the API module processes the data deletion request afterwards. Before migrating, the migration module first queries the destination resource pool and finds that there is no deletion mark column for the first metadata, so the first object data and the first metadata in the source resource pool are migrated. The API module writes the deletion mark column for the first metadata. At this time, the first object in the source resource pool has completed the migration, and there is a data column for the first metadata in the destination resource pool. The API module deletes the data column for the first metadata and the first object data. The migration system also includes a cleanup module. After the data migration is completed, there is a deletion mark column for the first metadata in the destination resource pool for the first object, and the cleanup module deletes the deletion mark column for the first metadata.

[0075] In one embodiment, in the migration system, the API module processes the data deletion request, and the concurrent migration module migrates the data. At this time, the first metadata has neither a data column nor a deletion mark column. The API module first writes the deletion mark column of the first metadata, and does not delete the data column of the first metadata and the first object data. The migration module migrates the first object data and the first metadata in the source resource pool. The migration system also includes a cleanup module. After the data migration is completed, the first object has the deletion mark column of the first metadata, the data column of the first metadata, and the first object data in the destination resource pool. The cleanup module deletes the deletion mark column of the first metadata, the data column of the first metadata, and the first object data.

[0076] In this embodiment, data migration can be achieved without affecting the API module's processing of data deletion requests, and strong consistency of data during the migration process can be ensured.

[0077] In an exemplary embodiment, the data read request is used to request to read the second object data and the second metadata of the second object. Figure 6 , step 201 reads data based on the source resource pool and the destination resource pool, including steps 601 to 603, wherein:

[0078] Step 601: If a deletion mark column of the second metadata exists in the destination resource pool, the output data reading result is that the second object does not exist.

[0079] Before reading the second object data and the second metadata, it is necessary to first check whether the second metadata and the second object data exist in the destination resource pool. If the second object has been deleted, there will be a deletion mark column of the second metadata in the destination resource pool. Therefore, if there is a deletion mark column of the second metadata in the destination resource pool, the output data reading result is that the second object does not exist.

[0080] Step 602: If there is no deletion mark column of the second metadata in the destination resource pool, and there is a data column of the second metadata in the destination resource pool or the source resource pool, read the second object data according to the data column of the second metadata, and output the data reading result as the second object data and the second metadata.

[0081] If there is no deletion mark column of the second metadata, it means that the second object has not been deleted. At this time, the second object data and the second metadata can be further queried in the destination resource pool or the source resource.

[0082] More specifically, if the second object is an object that has not been migrated in the source resource pool, the second object data and the second metadata need to be read from the source resource pool. If the second object is an object that has been migrated to the destination resource pool, the second metadata and the second object data can be directly read from the destination resource pool. If the second object is an object that is newly written to the destination resource pool, the second metadata and the second object data can be directly read from the destination resource pool.

[0083] Step 603: If the data column of the second metadata does not exist in the destination resource pool and the source resource pool, the output data reading result is that the second object does not exist.

[0084] If the data column of the second metadata does not exist in either the destination resource pool or the source resource pool, it is proved that the second object does not exist.

[0085] In one embodiment, since both the source resource pool and the destination resource pool support data reading operations, the API module can read data from the destination resource pool and the source resource pool when processing a data reading request.

[0086] Before reading data, it is necessary to determine whether the API module has deleted the second object by checking whether there is a deletion mark column in the second metadata of the destination resource pool. If there is no deletion mark column in the second metadata, it means that the API module has not deleted the second object. At this time, the metadata and data of the object can be further queried in the destination resource pool or the source resource pool. If the second object the user requests to read is an object that has not been migrated in the source resource pool, the API module needs to read the second metadata and second object data in the source resource pool. If the second object the user requests to read is an object in the source resource pool that has been migrated to the destination resource pool, the API module can directly read the second metadata and second object data in the destination resource pool. If the second object the user requests to read is an object newly written to the destination resource pool during migration, the API module can directly read the corresponding second metadata and second object data in the destination resource pool.

[0087] Specifically, query the second metadata of the second object in the destination resource pool to determine whether there is a deletion mark column in the second metadata. If there is a deletion mark column in the second metadata in the destination resource pool, it means that the second object has been deleted, and a reply is made that the second object does not exist. If there is no deletion mark column for the second metadata in the destination resource pool, determine whether there is a data column for the second metadata in the destination resource pool. If there is a data column for the second metadata in the destination resource pool, it means that the object exists in the destination resource pool, read the second object data according to the data column of the second metadata, and output the second object metadata and the second metadata. If there is no data column for the second metadata in the destination resource pool, query the second metadata in the source resource pool to determine whether there is a data column for the second metadata in the source resource pool. If there is a data column for the second metadata in the source resource pool, read the second object data according to the data column of the second metadata, and reply with the second metadata and the second object data. If there is no data column for the second metadata in the source resource pool, it means that the second object does not exist in the source resource pool either, and a reply is made that the second object does not exist.

[0088] In this embodiment, data migration can be achieved without affecting the processing of data read requests by the API module, and strong consistency of data during the migration process can be guaranteed.

[0089] In an exemplary embodiment, the data write request is used to request writing third object data and third metadata of a third object. Figure 7 Step 202 writes data based on the destination resource pool, including:

[0090] Step 701: If the original object data and original metadata of the third object exist in the destination resource pool, the third object data is written into the destination resource pool and the original object data is overwritten, and the third metadata is written into the destination resource pool and the original metadata is overwritten.

[0091] If the destination resource pool already has the third object data and the third metadata in the migration process, then writing the third object data and the third metadata is actually overwriting. At this time, the data migration writing may be overwritten by the concurrent terminal.

[0092] In the migration system, the API module modifies the existing third-party object data and third-party metadata, which is an overwrite. It is possible that the migration write is concurrent with the API overwrite.

[0093] In one embodiment, if the API module writes to a third object during data migration, a situation where the API module writes to the third object and concurrent migration does not occur.

[0094] In one embodiment, in the migration system, the API module first processes the data write request, and the migration module performs data migration afterwards. Specifically, the API module first writes the data columns of the third object data and the third metadata. Before the migration, the migration module first queries the destination resource pool and finds that the destination resource pool has the data column of the third object metadata, so there is no need to migrate the third metadata and the third object data in the source resource pool.

[0095] In one embodiment, in the migration system, the migration module migrates first, and the API module processes the data writing request later. Before the migration, the migration module queries the destination resource pool first, and finds that the data column of the third metadata does not exist in the destination resource pool, so the third object data and the third metadata in the source resource pool are migrated. The API module writes new third object data to the destination resource pool, and modifies the data column of the third metadata written by the migration module in the destination resource pool.

[0096] In one embodiment, in the migration system, the API module processes data write requests and the concurrent migration module migrates data. Specifically, the API module overwrites the writes and the concurrent migration module migrates the writes. The API module writes the data columns of object data and metadata to the destination resource pool in sequence. At the same time, the migration module detects that the data column of metadata does not exist in the destination resource pool and starts migrating the data columns of object data and metadata in the source resource pool. Since the migration module uses the CAS method to migrate the data column of metadata, it will check again whether the data column of metadata exists when writing metadata. At this time, the data column of metadata already exists, and the reply to migrate the data column of metadata fails. When the data migration is subsequently retried, the migration module detects that the data column of third metadata exists for the third object in the destination resource pool and does not migrate the third object.

[0097] In this embodiment, data migration can be achieved without affecting the processing of data write requests by the API module, and strong consistency of data during the migration process can be guaranteed.

[0098] For an exemplary embodiment, see Figure 8 , the source resource pool is constructed based on multiple servers. Before data migration, the data migration method further includes steps 801 to 802, wherein;

[0099] Step 801: Obtain the object to be migrated specified by each server, where the object to be migrated includes object data to be migrated and metadata to be migrated.

[0100] Specify different scan start and end row keys (RowKey) on different servers, and multiple servers jointly perform migration tasks. In the database scan operation, the start and end RowKeys are used to limit the scan range. RowKey is unique in the metadata table and is used to uniquely identify a row of metadata. Optionally, metadata is globally sorted and stored in the ASCII dictionary order of the row key.

[0101] Step 802: For each server, create a migration thread for the object to be migrated in the thread pool.

[0102] On a single server, set the start and end ranges of table scanning to scan the object metadata table of the source resource pool. Obtain all objects whose metadata RowKey value range is [StartRowKey, EndRowKey). Create a migration thread for each object in the thread pool to migrate the object data (object data to be migrated) and metadata (metadata to be migrated) of all objects within the metadata RowKey value range (i.e., the specified objects to be migrated).

[0103] Accordingly, the data migration process includes:

[0104] When there is no data column of the metadata to be migrated and no deletion mark column in the destination resource pool, the object data to be migrated and the metadata to be migrated in the original resource pool are migrated to the destination resource pool.

[0105] In one embodiment, after the migration thread is created, the destination resource pool is queried whether there is a data column or a deletion mark column for the metadata to be migrated. If there is a data column for the metadata to be migrated, it means that the object to be migrated already exists and no migration is required, and the migration is replied as completed. If there is a deletion mark column for the metadata to be migrated in the destination resource pool, it means that the API module has deleted the object to be migrated and no migration is required, and the migration is replied as completed. If there is no data column or deletion mark column for the metadata to be migrated in the destination resource pool, it means that the object to be migrated has neither been deleted nor migrated, and the data of the object to be migrated and the metadata to be migrated are migrated from the source resource pool to the destination resource pool.

[0106] For an exemplary embodiment, see Fig. 9 , the data migration method further includes:

[0107] Step 901, after completing the data migration, for the object data to be deleted and the metadata to be deleted included in the object to be deleted, if there is a deletion mark column of the metadata to be deleted in the destination resource pool, and there is a data column of the metadata to be deleted in the destination resource pool, then delete the data column of the metadata to be deleted, the deletion mark column of the metadata to be deleted, and the object data to be deleted.

[0108] The migration system also includes a cleanup module. After the migration is completed, the API module is switched to the API of the destination resource pool, and the deletion mark column in the destination resource pool is no longer increased. The cleanup module is responsible for clearing the deletion mark column written by the API module during the migration process, as well as the redundant metadata data column and redundant object data generated by the concurrent operation of the API module deleting data and the migration module migrating data. Different table scan start and end RowKeys are specified on different servers, and multiple servers jointly perform the task of cleaning up the deletion mark. On a single server, the table scan start and end range is set to scan the object metadata table of the destination resource pool, and all objects (i.e., objects to be deleted) with a metadata Rowkey value range of [StartRowKey, EndRowKey) are obtained, and a cleanup thread is created for each object in the thread pool. Query whether there is a deletion mark column for the metadata to be deleted in the destination resource pool. If there is no deletion mark column for the metadata to be deleted, end the cleanup thread. If there is a deletion mark column for the metadata, query whether there is a data column for the metadata to be deleted in the destination resource pool. If there are data columns in the destination resource pool, it means that redundant metadata data columns and redundant object data are generated due to the concurrent operations of API module deletion and migration module migration. Delete the object metadata deletion mark column, data column and data of the destination resource pool, and end the cleanup thread.

[0109] Step 902: If there is a deletion mark column of the metadata to be deleted in the destination resource pool, and there is no data column of the metadata to be deleted in the resource pool, then the deletion mark column of the metadata to be deleted is deleted.

[0110] If the data column of the metadata to be deleted does not exist in the destination resource pool, it means that the destination resource pool does not have the object data to be deleted, and the deletion mark column of the metadata to be deleted in the destination resource pool is deleted, and the cleaning thread ends.

[0111] The method provided in this embodiment can ensure strong consistency of data in the source resource pool and the destination resource pool during the migration process by deleting data through the cleaning module.

[0112] It should be understood that, although the steps in the flowcharts of the embodiments described above are shown in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0113] Based on the same inventive concept, the embodiment of the present application also provides a data migration device for implementing the data migration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data migration device embodiments provided below can refer to the limitations on the data migration method above, and will not be repeated here.

[0114] In an exemplary embodiment, Fig.10 As shown, a data migration device 100 is provided, comprising: a data reading module 1001, a data writing module 1002 and a data deleting module 1003, wherein:

[0115] The data reading module 1001 is used to read data based on the source resource pool and the destination resource pool in response to a data reading request sent by the terminal during the data migration process.

[0116] The data writing module 1002 is used to write data based on the destination resource pool in response to a data writing request sent by the terminal during the data migration process.

[0117] The data deletion module 1003 is used to delete data based on the destination resource pool in response to a data deletion request sent by the terminal during data migration; wherein the source resource pool only supports reading operations of object data and metadata, and the destination resource pool supports reading operations, writing operations and deleting operations of object data and metadata, and the metadata is used to describe the object data; wherein data migration is used to migrate object data and metadata in the source resource pool to the destination resource pool.

[0118] In one embodiment, the data deletion request is used to request deletion of first object data and first metadata of a first object; the data deletion module 1003 is used to: write a deletion mark column of the first metadata in a destination resource pool; if data columns of the first object data and the first metadata exist in the destination resource pool, then based on the deletion mark column of the first metadata, delete the first object data and the first metadata; if data columns of the first object data and the first metadata do not exist in the destination resource pool, then during the data migration process, the first object data and the second metadata are not migrated.

[0119] In one embodiment, the data read request is used to request reading of second object data and second metadata of a second object; the data reading module 1001 is used to: if there is a deletion mark column of the second metadata in the destination resource pool, then output the data reading result as that the second object does not exist; if there is no deletion mark column of the second metadata in the destination resource pool, and there is a data column of the second metadata in the destination resource pool or the source resource pool, then read the second object data according to the data column of the second metadata, and output the data reading result as the second object data and the second metadata; if there is no data column of the second metadata in both the destination resource pool and the source resource pool, then output the data reading result as that the second object does not exist.

[0120] In one embodiment, the data write request is used to request writing of third object data and third metadata of a third object; the data write module 1002 is used to: if the original object data and original metadata of the third object exist in the destination resource pool, write the third object data to the destination resource pool and overwrite the original object data, and write the third metadata to the destination resource pool and overwrite the original metadata.

[0121] In one embodiment, the source resource pool is constructed based on multiple servers; before the data migration, the data migration device 12 also includes a processing module 1004, which is used to obtain the object to be migrated specified by each server, and the object to be migrated includes the object data to be migrated and the metadata to be migrated; for each server, a migration thread for the object to be migrated is created in the thread pool; when there is no data column for the metadata to be migrated and no deletion mark column in the destination resource pool, the object data to be migrated and the metadata to be migrated in the original resource pool are migrated to the destination resource pool.

[0122] In one embodiment, the processing module 1004 is also used for: after completing the data migration, for the object data to be deleted and the metadata to be deleted included in the object to be deleted, if there is a deletion mark column for the metadata to be deleted in the destination resource pool, and there is a data column for the metadata to be deleted in the destination resource pool, then the data column for the metadata to be deleted, the deletion mark column for the metadata to be deleted, and the object data to be deleted are deleted; if there is a deletion mark column for the metadata to be deleted in the destination resource pool, and there is no data column for the metadata to be deleted in the resource pool, then the deletion mark column for the metadata to be deleted is deleted.

[0123] Each module in the above-mentioned data migration device 100 can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0124] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.11 As shown. The computer device includes a processor, a memory, an input / reply interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / reply interface are connected through a system bus, and the communication interface is connected to the system bus through the input / reply interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store metadata, object data, etc. The input / reply interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data migration method is implemented.

[0125] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data migration method provided in any of the above embodiments is implemented.

[0127] In one embodiment, a computer program product is provided, including a computer program, which implements the data migration method provided in any of the above embodiments when executed by a processor.

[0128] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0129] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0130] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data migration method, characterized in that: The method comprises: During the data migration process, in response to a data read request sent by the terminal, data is read based on the source resource pool and the destination resource pool; During the data migration process, in response to a data write request sent by the terminal, data is written based on the destination resource pool; During the data migration process, data deletion is performed based on the destination resource pool in response to a data deletion request sent by the terminal; wherein the source resource pool only supports reading operations of object data and metadata, and the destination resource pool supports reading operations, writing operations, and deleting operations of object data and metadata, and the metadata is used to describe the object data; wherein data migration is used to migrate object data and metadata in the source resource pool to the destination resource pool.

2. The method according to claim 1, characterized in that The data deletion request is used to request deletion of first object data and first metadata of a first object; and the data deletion based on the destination resource pool includes: Writing a deletion mark column of the first metadata into the destination resource pool; If the data columns of the first object data and the first metadata exist in the destination resource pool, deleting the first object data and the first metadata based on the deletion mark column of the first metadata; If the data columns of the first object data and the first metadata do not exist in the destination resource pool, the first object data and the second metadata are not migrated during the data migration process.

3. The method according to claim 1, characterized in that The data reading request is used to request to read the second object data and the second metadata of the second object; the data reading based on the source resource pool and the destination resource pool includes: If the destination resource pool contains a deletion mark column for the second metadata, the output data reading result is that the second object does not exist; If the destination resource pool does not contain a deletion mark column of the second metadata, and the destination resource pool or the source resource pool contains a data column of the second metadata, the second object data is read according to the data column of the second metadata, and the data reading result is output as the second object data and the second metadata; If the data column of the second metadata does not exist in the destination resource pool and the source resource pool, the output data reading result is that the second object does not exist.

4. The method according to claim 1, characterized in that: The data writing request is used to request writing third object data and third metadata of the third object; and the data writing based on the destination resource pool includes: If the original object data and original metadata of the third object exist in the destination resource pool, the third object data is written into the destination resource pool and the original object data is overwritten, and the third metadata is written into the destination resource pool and the original metadata is overwritten.

5. The method according to any one of claims 1 to 4, characterized in that: The source resource pool is constructed based on multiple servers; before the data migration, the method further includes: Obtaining an object to be migrated specified by each server, wherein the object to be migrated includes object data to be migrated and metadata to be migrated; For each server, create a migration thread for the object to be migrated in the thread pool; Accordingly, the data migration process includes: When there is no data column of the metadata to be migrated and no deletion mark column in the destination resource pool, the object data to be migrated and the metadata to be migrated in the original resource pool are migrated to the destination resource pool.

6. The method according to claim 5, characterized in that The method further comprises: After completing the data migration, for the object data to be deleted and the metadata to be deleted included in the object to be deleted, if there is a deletion mark column of the metadata to be deleted in the destination resource pool, and there is a data column of the metadata to be deleted in the destination resource pool, then delete the data column of the metadata to be deleted, the deletion mark column of the metadata to be deleted, and the object data to be deleted; If the destination resource pool has a deletion marker column of the metadata to be deleted, and the resource pool does not have a data column of the metadata to be deleted, the deletion marker column of the metadata to be deleted is deleted.

7. A data migration device, characterized in that: The device comprises: A data reading module is used to read data based on a source resource pool and a destination resource pool in response to a data reading request sent by a terminal during data migration; A data writing module, used for responding to a data writing request sent by a terminal during data migration and writing data based on the destination resource pool; A data deletion module is used to delete data based on the destination resource pool in response to a data deletion request sent by a terminal during data migration; wherein the source resource pool only supports reading operations of object data and metadata, and the destination resource pool supports reading operations, writing operations, and deleting operations of object data and metadata, and the metadata is used to describe the object data; wherein data migration is used to migrate object data and metadata in the source resource pool to the destination resource pool.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images