Data migration method and system
By designing a data migration system in cloud computing, using storage interface modules to isolate applications and source storage instances, cache and forward data operation requests, the application interruption problem caused by data migration in cloud computing is solved, and efficient and uninterrupted data migration is achieved process.
Patent Information
- Application Number
- CN202311041638.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-08
- Filing Date
- 2023-08-17
- Publication Date
- 2024-11-08
AI Technical Summary
In cloud computing scenarios, application interruptions often occur during data migration, and existing cloud services such as RDS and DMS cannot avoid interrupt problems during migration.
A data migration method and system are designed to isolate the application from the source storage instance through the storage interface module, cache data operation requests, and send requests to the target storage instance during the migration process to ensure that the application does not interrupt. The system includes a migration module, a cache module and a storage interface module, and the management module supports tenant configuration and real-time monitoring of migration status.
Complete data migration without interrupting the application, supports the migration of heterogeneous databases and table-based scenarios, and improves the efficiency and controllability of data migration.
Smart Images

Figure CN118916341A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of cloud computing technology, and in particular to a data migration method and system. Background Art
[0002] In cloud computing scenarios, as applications change, the backend storage services (e.g., databases) may need data migration. To this end, some cloud service providers provide related cloud services, such as relational database service (RDS) and database migration service (DMS). However, using the above cloud services for data migration will cause application interruption. Summary of the invention
[0003] The present application provides a data migration method and system, which can realize data migration of its backend storage service without interrupting the application.
[0004] In a first aspect, the present application provides a data migration method. The method can be used in a data migration system. The data migration system includes a migration module, a cache module, and a storage interface module. The storage interface module receives a data operation request from an application, sends the data operation request to a source storage instance, and caches the data operation request to the cache module. The migration module migrates data in the source storage instance to a target storage instance, and sends the data operation request in the cache module to the target storage instance. The storage interface module stops sending new data operation requests from the business to the source storage instance, and sends the new data operation request to the target storage instance.
[0005] In the technical solution provided by the present application, the data migration system can achieve isolation between the application and the source storage instance through the storage interface module, and in the process of migrating the data in the source storage instance to the target storage instance, the storage interface module can send the data operation request from the application to the source storage instance, so that the application can still perform write operations on the source storage instance, thus ensuring that the application will not be interrupted during the data migration process. In other words, through the above data migration system, the data in the source storage instance can be migrated to the target storage instance without interrupting the application.
[0006] In a possible implementation of the first aspect, the method further includes: when a data operation request in the cache module meets a preset condition, the storage interface module suspends a new data operation request. The preset condition includes any one or more of the following: the number of data operation requests in the cache module is less than a first threshold, and the amount of data in the data operation request in the cache module is less than a second threshold. Suspending a new data operation request means that after the storage interface module receives a new data operation request from an application, the data operation request is not sent to the source storage instance, nor is the data operation request cached to the cache module. It should be understood that when the data operation request in the cache module meets the preset condition, it means that there are very few data operation requests in the current cache module, and the time that the new data operation request is suspended will be very short. Since the storage interface module isolates the application and the source storage instance, the application is not interrupted.
[0007] In a possible implementation of the first aspect, the storage interface module sends the new data operation request to the target storage instance, including: when the number of data operation requests and / or the amount of data in the cache module is 0, the storage interface module sends the new data operation request to the target storage instance. It should be understood that when the number of data operation requests and / or the amount of data in the cache module is 0, it means that all data in the source storage instance has been migrated to the target storage instance. At this time, the target storage instance can replace the source storage instance, so the storage interface module can directly send the new data operation request to the target storage instance.
[0008] In a possible implementation of the first aspect, the source storage instance and the target storage instance are heterogeneous databases. That is, the technical solution provided by the present application supports data migration between heterogeneous databases.
[0009] In a possible implementation of the first aspect, the data migration system further includes a management module. The management module supports receiving the configuration of the tenant, and supports displaying the status of data migration to the tenant. Wherein, the configuration of the tenant includes any one or more of the following: the identifier of the source storage instance, the identifier of the target storage instance, and the data migration strategy. The data migration strategy indicates any one or more of the following operations: converting the format of the data in the source storage instance to the format supported by the target storage instance, and migrating the data after the format conversion to the target storage instance; converting the format of the data operation request in the cache module to the format supported by the target storage instance, and sending the data operation request after the format conversion to the target storage instance; converting the format of the new data operation request to the format supported by the target storage instance, and sending the new data operation request after the format conversion to the target storage instance. The status of data migration refers to the completion of this data migration task, for example, the number or amount of data operation requests in the cache module, how much data has been migrated from the source storage instance, how long it is expected to take to complete the data migration, etc. Through the above implementation, tenants can flexibly configure data migration tasks according to actual business needs, and can also view the progress of data migration tasks in real time.
[0010] In a possible implementation of the first aspect, the storage interface module sends a new data operation request to the target storage instance, including: the storage interface module converts the format of the new data operation request into a format supported by the target storage instance according to the data migration strategy, and sends the new data operation request after the format conversion to the target storage instance. It should be understood that in the above implementation, the storage interface module has the adaptation and conversion functions of the database protocol, so that data migration between heterogeneous databases can be realized.
[0011] In a second aspect, the present application provides another data migration method. The method can be used in a data migration system. The data migration system includes a migration module, a cache module, and a storage interface module. The storage interface module receives a data operation request from an application, sends the data operation request to a source storage instance, and caches the data operation request to the cache module. The migration module migrates data in the source storage instance to a target storage instance, and sends the data operation request in the cache module to the target storage instance. The storage interface module sends new data operation requests from the business to the source storage instance and the target storage instance.
[0012] In the technical solution provided by the present application, the data migration system can achieve isolation between the application and the source storage instance through the storage interface module, and in the process of migrating the data in the source storage instance to the target storage instance, the storage interface module can send the data operation request from the application to the source storage instance, so that the application can still perform write operations on the source storage instance, thus ensuring that the application will not be interrupted during the data migration process. In other words, through the above data migration system, the data in the source storage instance can be migrated to the target storage instance without interrupting the application.
[0013] In a possible implementation of the second aspect, the method further includes: when a data operation request in the cache module meets a preset condition, the storage interface module suspends a new data operation request. The preset condition includes any one or more of the following: the number of data operation requests in the cache module is less than a first threshold, and the amount of data in the data operation request in the cache module is less than a second threshold. Suspending a new data operation request means that after the storage interface module receives a new data operation request from an application, the data operation request is not sent to the source storage instance, nor is the data operation request cached to the cache module. It should be understood that when the data operation request in the cache module meets the preset condition, it means that there are very few data operation requests in the current cache module, and the time that the new data operation request is suspended will be very short. Since the storage interface module isolates the application and the source storage instance, the application is not interrupted.
[0014] In a possible implementation of the second aspect, the target storage instance is a sub-table of the source storage instance. That is, the technical solution provided by the present application supports data migration between sub-table scenarios.
[0015] In a possible implementation of the second aspect, the storage interface module sends the new data operation request to the source storage instance and the target storage instance, including: when the number of data operation requests and / or the amount of data in the cache module is 0, the storage interface module sends the new data operation request to the source storage instance and the target storage instance. It should be understood that when the number of data operation requests and / or the amount of data in the cache module is 0, it means that all the data in the source storage instance that meets the migration requirements has been migrated to the target storage instance, that is, the splitting of the source storage instance has been completed. Therefore, the storage interface module can directly send the new data operation request to the source storage instance and the target storage instance.
[0016] In a possible implementation of the second aspect, the data migration system further includes a management module. The management module supports receiving the tenant's configuration, and supports displaying the status of data migration to the tenant. Among them, the tenant's configuration includes any one or more of the following: migrating data that meets the migration requirements in the source storage instance to the target storage instance, sending data operation requests that meet the migration requirements in the cache module to the target storage instance, and sending new data operation requests that meet the migration requirements to the target storage instance. The status of data migration refers to the completion status of this data migration task, for example, the number or amount of data operation requests in the cache module, how much data has been migrated from the source storage instance, how long it is expected to take to complete the data migration, etc. Through the above implementation, tenants can flexibly configure data migration tasks according to actual business needs, and can also view the progress of data migration tasks in real time.
[0017] In a third aspect, the present application provides a data migration system. The system includes a storage interface module, a cache module and a migration module. The storage interface module is used to receive a data operation request from an application, send the data operation request to a source storage instance, cache the data operation request to the cache module, and stop sending new data operation requests to the source storage instance, and send new data operation requests to the target storage instance. The cache module is used to cache the above data operation requests. The migration module is used to migrate data in the source storage instance to the target storage instance, and send the data operation request in the cache module to the target storage instance.
[0018] In a possible implementation of the third aspect, the storage interface module is further configured to suspend new data operation requests when a data operation request in the cache module meets a preset condition. The preset condition includes any one or more of the following: the number of data operation requests in the cache module is less than a first threshold, and the amount of data in the data operation request in the cache module is less than a second threshold. Suspending new data operation requests means that after receiving a new data operation request from an application, the storage interface module does not send the data operation request to the source storage instance, nor does it cache the data operation request to the cache module.
[0019] In a possible implementation manner of the third aspect, the storage interface module is used to send a new data operation request to the target storage instance when the number of data operation requests and / or the amount of data in the cache module is 0.
[0020] In a possible implementation manner of the third aspect, the source storage instance and the target storage instance are heterogeneous databases.
[0021] In a possible implementation of the third aspect, the data migration system further includes a management module. The management module is used to receive the tenant's configuration and display the status of data migration to the tenant. The tenant's configuration includes any one or more of the following: the identifier of the source storage instance, the identifier of the target storage instance, and the data migration strategy. The data migration strategy indicates any one or more of the following operations: converting the format of the data in the source storage instance to a format supported by the target storage instance, and migrating the data after the format conversion to the target storage instance; converting the format of the data operation request in the cache module to a format supported by the target storage instance, and sending the data operation request after the format conversion to the target storage instance; converting the format of the new data operation request to a format supported by the target storage instance, and sending the new data operation request after the format conversion to the target storage instance. The status of data migration refers to the completion status of this data migration task, for example, the number or amount of data operation requests in the cache module, how much data has been migrated from the source storage instance, how long it is expected to take to complete the data migration, etc.
[0022] In a possible implementation of the third aspect, the storage interface module is used to convert the format of the new data operation request into a format supported by the target storage instance according to the data migration strategy, and send the new data operation request after the format conversion to the target storage instance.
[0023] In a fourth aspect, the present application provides another data migration system. The system includes a storage interface module, a cache module and a migration module. The storage interface module is used to receive a data operation request from an application, send the data operation request to a source storage instance, cache the data operation request to the cache module, and send a new data operation request to the source storage instance and the target storage instance. The cache module is used to cache the above data operation request. The migration module is used to migrate data in the source storage instance to the target storage instance, and send the data operation request in the cache module to the target storage instance.
[0024] In a possible implementation of the fourth aspect, the storage interface module is further configured to suspend new data operation requests when a data operation request in the cache module meets a preset condition. The preset condition includes any one or more of the following: the number of data operation requests in the cache module is less than a first threshold, and the amount of data in the data operation request in the cache module is less than a second threshold. Suspending new data operation requests means that after receiving a new data operation request from an application, the storage interface module does not send the data operation request to the source storage instance, nor does it cache the data operation request to the cache module.
[0025] In a possible implementation of the fourth aspect, the storage interface module is used to send a new data operation request to the source storage instance and the target storage instance when the number of data operation requests and / or the amount of data in the cache module is 0.
[0026] In a possible implementation manner of the fourth aspect, the target storage instance is a sub-table of the source storage instance.
[0027] In a possible implementation of the fourth aspect, the above-mentioned data migration system also includes a management module. The management module is used to receive the tenant's configuration and display the status of data migration to the tenant. Among them, the tenant's configuration includes any one or more of the following: the identifier of the source storage instance, the identifier of the target storage instance, and the data migration policy. The data migration policy indicates any one or more of the following operations: migrating the data that meets the migration requirements in the source storage instance to the target storage instance, sending the data operation request that meets the migration requirements in the cache module to the target storage instance, and sending the new data operation request that meets the migration requirements to the target storage instance. The status of data migration refers to the completion status of this data migration task, for example, the number of data operation requests or the amount of data in the cache module, how much data has been migrated from the source storage instance, how long it is expected to take to complete the data migration, etc.
[0028] In a fifth aspect, the present application provides a computing device. The computing device includes a processor and a memory. The processor is used to execute instructions stored in the memory to implement part or all of the methods described in the first aspect and any implementation thereof, the second aspect and any implementation thereof.
[0029] In a sixth aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, and each computing device includes a processor and a memory. The processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster performs part or all of the methods described in the first aspect and any implementation thereof, the second aspect and any implementation thereof.
[0030] In a seventh aspect, the present application provides a computer program product. The computer program product may be software or a program product that includes instructions and can be run on a computing device or stored in any available medium. When the computer program product is run on a computing device or a computing device cluster, the computing device or the computing device cluster executes part or all of the methods described in the aforementioned first aspect and any implementation thereof, and the second aspect and any implementation thereof.
[0031] In an eighth aspect, the present application provides a computer-readable storage medium. The computer storage medium includes computer program instructions, and when the computer program instructions are executed by a computing device or a computing device cluster, the computing device or the computing device cluster executes part or all of the methods described in the aforementioned first aspect and any implementation thereof, and the second aspect and any implementation thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present application;
[0033] Figure 2 is a schematic diagram of another application scenario provided by an embodiment of the present application;
[0034] Figure 3 It is a structural diagram of a data migration system provided in an embodiment of the present application;
[0035] Figure 4 It is a flowchart of a data migration method provided in an embodiment of the present application;
[0036] Figure 5 It is a flowchart of another data migration method provided in an embodiment of the present application;
[0037] Figure 6 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;
[0038] Figure 7 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;
[0039] Figure 8 It is a structural diagram of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION
[0040] The technical solution provided by the present application is described below in conjunction with the accompanying drawings.
[0041] Storage services (e.g., databases) are essential components in cloud service development. They are used to store various data required by applications. Therefore, as applications change, their backend storage services also need to be adjusted accordingly. For example, when the amount of application data increases dramatically, it is necessary to migrate the data in the current database to another database with a larger specification, or migrate part of the data in the current database to another database to alleviate the storage pressure of the current database. For another example, when an application places higher requirements on data security, it is necessary to migrate part or all of the data in the current database to another database with higher security.
[0042] To facilitate data migration, some cloud service providers provide related cloud services, such as RDS and DMS. RDS is scalable (that is, it can automatically expand storage capacity according to the growth of storage demand), and RDS also supports migrating databases from one storage instance (for example, a virtual machine or container) to another storage instance. DMS supports migrating heterogeneous databases to RDS. However, in the process of using RDS and DMS for data migration, both will cause application interruption.
[0043] In response to the above problems, an embodiment of the present application provides a data migration system that can complete data migration of its backend storage service without interrupting the application.
[0044] The data migration system provided by the embodiment of the present application is flexible in deployment. For example, the data migration system can be independently deployed in a data center of a cloud environment, and the data center of the cloud environment includes a large number of basic resources (including computing resources, storage resources, and network resources), and the computing resources in the data center of the cloud environment can be a large number of computing devices (for example, servers). Specifically, the data migration system can be independently deployed on a server or a virtual machine in a data center of a cloud environment, or distributedly deployed on multiple servers or multiple virtual machines in a data center of a cloud environment, or distributedly deployed on servers and virtual machines in a data center of a cloud environment.
[0045] In some embodiments, the cloud service provider provides the functions of the data migration system to the tenant as part of the business service. Figure 1 As shown, when a tenant wants to implement a certain application (for example, an artificial intelligence application, a big data business, etc.), he can register an account on the cloud management platform, log in to the cloud management platform with the account, and then purchase the corresponding business service on the cloud management platform (for example, he can pre-charge and then settle according to the final resource usage, or purchase it on an annual or monthly basis). After the purchase is successful, the tenant can use the cloud service to implement the above application. When using the cloud service, if it involves a scenario where the data in the database of the application backend needs to be migrated to another database (for example, the specifications or security of the database of the current application backend cannot meet business needs), the data migration system deployed in the data center of the cloud environment will complete the above data migration without interrupting the application.
[0046] In some other embodiments, the cloud service provider may also provide the functions of the data migration system to the tenant as a separate cloud service. Figure 2As shown, when a tenant wants to migrate data in the application backend database to another database, the tenant can register an account on the cloud management platform, log in to the cloud management platform through the account, and purchase the cloud service on the cloud management platform (for example, pre-charge and then settle according to the final resource usage, or purchase on an annual or monthly basis). After the purchase is successful, the tenant can use the cloud service to migrate data in the current application backend database to another database.
[0047] For another example, the data migration system may also be deployed on one or more terminal computing devices, including terminal servers, smart phones, laptops, tablet computers, personal desktop computers, etc. Alternatively, the data migration system may also be deployed in a distributed manner in different environments, for example, a portion of the data migration system may be deployed on a computing device in a data center of a cloud environment, and another portion may be deployed on a terminal computing device, and the various portions of the data migration system deployed in different environments may collaboratively provide data migration functions.
[0048] It should be understood that the present application embodiment does not limit the division of the various parts of the data migration system (for example, it can be Figure 3 The division method shown in the figure does not limit the deployment method of each part of the data migration system. In actual application, it can be adaptively deployed according to specific application requirements or the computing power of hardware equipment.
[0049] Figure 3 The structural diagram of the above data migration system is shown as an example. Figure 3 As shown, the data migration system 100 is used to implement data migration from the source storage instance 300 to the target storage instance 400 without interrupting the application 200. Among them, the application 200 can be a user-side service, which provides corresponding services to the outside through a gateway or load balancing. The source storage instance 300 refers to a storage instance that provides storage services for the application 200 before the start of the data migration task. For example, the application 200 can write the intermediate data and result data generated during the execution process to the source storage instance 300, and can also read the data required by the application 200 during the execution process from the source storage instance 300. Both the source storage instance 300 and the target storage instance 400 can include any one or more of the following: relational databases such as MySQL and PostgreSQL or data tables in relational databases; non-relational databases such as MongoDB, Hbase, ElasticSearch, or data tables in non-relational databases; other storage services, such as file storage services or key-value (KV) storage services, etc.
[0050] The data migration system 100 includes a storage interface module 110, a cache module 120 and a migration module 130. Optionally, the data migration system 100 also includes a management module 140. The following briefly introduces each part of the data migration system 100.
[0051] The storage interface module 110 is used to provide a simulated interface of the source storage instance 300, so the application 200 can access the source storage instance 300 through the module. Specifically, when the application 200 accesses the source storage instance 300, the application 200 sends an access request to the storage interface module 110. Accordingly, the storage interface module 110 receives the access request from the application 200 and sends the access request to the source storage instance 300, thereby enabling the application 200 to access the source storage instance 300. Among them, the access request includes a read request and / or a data operation request. Therefore, the application 200's access to the source storage instance 300 includes the application 200 reading data from the source storage instance 300, and / or the application 200 performing a write operation (including writing data, deleting data, and modifying data) on the source storage instance 300. It is worth noting that the embodiment of the present application does not limit the scenario in which the application 200 accesses the source storage instance 300 through the storage interface module 110. In other words, the application 200 can access the source storage instance 300 through the storage interface module 110 at any time, and can also access the source storage instance 300 through the storage interface module 110 only when data migration is required.
[0052] In the scenario where the source storage instance 300 migrates data to the target storage instance 400, considering that during the data migration process, the application 200 may initiate a data operation request to the source storage instance 300, and the data operation request may cause changes in the data in the source storage instance 300, the storage interface module 110 is also used to cache the data operation request received from the application 200 during the data migration process to the cache module 120. The data operation request in the cache module 120 will be subsequently sent to the target storage instance 400, thereby ensuring the data consistency between the source storage instance 300 and the target storage instance 400 during the data migration process.
[0053] The cache module 120 is used to cache the data operation request sent by the storage interface module 110 from the application 200. As an example, the cache module 120 can be a message queue, such as a Kafka queue. Further, for this data migration task, a topic can be created in the Kafka queue, and the topic is used to cache the data operation request sent by the storage interface module 110 from the application 200 in this data migration task.
[0054] The migration module 130 is used to migrate data in the source storage instance 300 to the target storage instance 400 , and to send data operation requests in the cache module 120 to the target storage instance 400 .
[0055] The management module 140 is used to receive the configuration of the tenant, so that the tenant can configure the data migration task according to actual needs. The management module 140 is also used to show the status of data migration to the tenant, for example, the number of data operation requests or data volume in the cache module 120, how much data has been migrated from the source storage instance 300, how long it is expected to take to complete the data migration task, etc. As an example, the management module 140 can implement the above functions by providing an access interface (for example, an application programming interface (API) or a graphical user interface (GUI)). In addition, the management module 140 is also used to manage the storage interface module 110 and the migration module 130, so that the management module 140 controls the execution of the entire data migration task, reduces the call chain between other modules during the data migration process, and improves the efficiency and success rate of data migration.
[0056] In the embodiment of the present application, the storage interface module 110, the cache module 120, the migration module 130 and the management module 140 can be implemented by software or by hardware. Exemplarily, the implementation of the storage interface module 110 is described by taking the storage interface module 110 as an example. Similarly, the implementation of the cache module 120, the migration module 130 and the management module 140 can refer to the implementation of the storage interface module 110.
[0057] As an example of a software functional unit, the storage interface module 110 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the storage interface module 110 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.
[0058] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.
[0059] As an example of a hardware functional unit, the storage interface module 110 may include at least one computing device (e.g., a server). Alternatively, the storage interface module 110 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). Among them, the PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0060] The multiple computing devices included in the storage interface module 110 can be distributed in the same region or in different regions. The multiple computing devices included in the storage interface module 110 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the storage interface module 110 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0061] Combine the following Figure 4-Figure 5 , describes in detail how the data migration system 100 implements data migration from the source storage instance 300 to the target storage instance 400 without interrupting the application 200.
[0062] First see Figure 4 , Figure 4 The following is a flow chart of a data migration method provided by an embodiment of the present application. Figure 4In the described embodiment, all data in the source storage instance 300 will be migrated to the target storage instance 400, and the source storage instance 300 and the target storage instance 400 can be heterogeneous storage instances or homogeneous storage instances. Among them, homogeneous storage instances refer to the source storage instance 300 and the target storage instance 400 having the same structure and external protocols. The structure of the storage instance describes the structure and layout of the data in the storage instance. Taking the database as an example, the structure of the database may include data tables, records, fields, and indexes. As an example, when the source storage instance 300 and the target storage instance 400 are both MySQL, the source storage instance 300 and the target storage instance 400 are homogeneous storage instances (also referred to as homogeneous databases herein). On the contrary, heterogeneous storage instances refer to the source storage instance 300 and the target storage instance 400 having inconsistent structures and / or external protocols. For example, when the source storage instance 300 is MongoDB and the target storage instance 400 is ElasticSearch, the source storage instance 300 and the target storage instance 400 are heterogeneous storage instances (also referred to as heterogeneous databases herein).
[0063] S101: The management module 140 receives the configuration of the tenant.
[0064] Specifically, since the management module 140 can provide an access interface (eg, API or GUI), the tenant can configure the data migration task through the interface, and accordingly, the management module 140 can receive the tenant's configuration through the interface.
[0065] The configuration of the tenant may include any one or more of the identifier of the source storage instance 300, the identifier of the target storage instance 400, and the data migration policy. The identifier of the source storage instance 300 is used to indicate the source storage instance 300. For example, the identifier of the source storage instance 300 may be the name, specification, and other information of the source storage instance 300 that can be used to identify the source storage instance 300. Similarly, the identifier of the target storage instance 400 is used to indicate the target storage instance 400. For example, the identifier of the target storage instance 400 may be the name, specification, and other information of the target storage instance 400 that can be used to identify the target storage instance 400. When the source storage instance 300 and the target storage instance 400 are isomorphic storage instances, the data migration policy is used to indicate any one or more of the following operations: migrate all data in the source storage instance 300 to the target storage instance 400, send all data operation requests in the cache module 120 to the target storage instance 400, and send all new data operation requests to the target storage instance 400. When the source storage instance 300 and the target storage instance 400 are heterogeneous storage instances, the data migration strategy is used to indicate any one or more of the following operations: converting the format of each data in the source storage instance 300 (i.e., the schema of the data, such as the data type, length or value range, etc.) into a format supported by the target storage instance 400, and migrating the data after the format conversion to the target storage instance 400; converting the format of each data operation request in the cache module 120 and the format of each new data operation request into a format supported by the target storage instance 400, and sending the data operation request after the format conversion to the target storage instance 400.
[0066] Optionally, any one or more of the above configurations may be configured by the tenant, or may be selected by the tenant based on a recommendation of the management module 140 , or may be configured by the management module 140 based on actual conditions.
[0067] S102 : The management module 140 sends a notification message A to the storage interface module 110 and a notification message B to the migration module 130 according to the tenant configuration. Accordingly, the storage interface module 110 receives the notification message A sent by the management module 140 , and the migration module 130 receives the notification message B sent by the management module 140 .
[0068] S103 : the storage interface module 110 receives a data operation request from the application 200 .
[0069] Specifically, the application 200 sends a data operation request to the storage interface module 110, and correspondingly, the storage interface module 110 receives the data operation request from the application 200. The data operation request in this step is a data operation request initiated by the application 200 to the source storage instance 300, which is used to instruct to perform a write operation on the source storage instance 300, for example, to write new data to the source storage instance 300, delete data in the source storage instance 300, and modify data in the source storage instance 300.
[0070] S104 : The storage interface module 110 sends the data operation request to the source storage instance 300 , and caches the data operation request to the cache module 120 .
[0071] Specifically, the storage interface module 110 sends the data operation request to the source storage instance 300. Accordingly, the source storage instance 300 receives the data operation request sent by the storage interface module 110 and performs updates (for example, adding data, deleting data, or modifying data) according to the data operation request. The storage interface module 110 sends the data operation request to the cache module 120 according to the notification message A. Accordingly, the cache module 120 receives the data operation request sent by the storage interface module 110 and caches the data operation request.
[0072] S105 : The migration module 130 migrates the data in the source storage instance 300 to the target storage instance 400 .
[0073] Specifically, the migration module 130 obtains the identifier of the source storage instance 300 , the identifier of the target storage instance 400 , and the data migration policy according to the notification message B, and then migrates the data in the source storage instance 300 to the target storage instance 400 according to the data migration policy.
[0074] Further, when the source storage instance 300 and the target storage instance 400 are homogeneous storage instances, the migration module 130 migrates the data in the source storage instance 300 to the target storage instance 400 according to the data migration strategy, including: the migration module 130 traverses the source storage instance 300, and copies each piece of data in the source storage instance 300 to the target storage instance 400. When the source storage instance 300 and the target storage instance 400 are heterogeneous storage instances, the migration module 130 migrates the data in the source storage instance 300 to the target storage instance 400 according to the data migration strategy, including: the migration module 130 traverses the source storage instance 300, and converts the format of each piece of data in the source storage instance 300 into a format supported by the target storage instance 400, and then copies each piece of data after the format conversion to the target storage instance 400.
[0075] S106 : The migration module 130 sends the data operation request in the cache module 120 to the target storage instance 400 .
[0076] Specifically, when condition A is met, the migration module 130 sends the data operation request in the cache module 120 to the target storage instance 400 according to the data migration strategy. Accordingly, the target storage instance 400 receives the data operation request sent by the migration module 130 and updates (for example, adds data, deletes data, or modifies data) according to the data operation request. It should be understood that when condition A is not met, the migration module 130 continues to execute the above S105.
[0077] Further, when the source storage instance 300 and the target storage instance 400 are homogeneous storage instances, the migration module 130 sends the data operation request in the cache module 120 to the target storage instance 400 according to the data migration strategy, including: the migration module 130 sends each data operation request in the cache module 120 to the target storage instance 400. When the source storage instance 300 and the target storage instance 400 are heterogeneous storage instances, the migration module 130 sends the data operation request in the cache module 120 to the target storage instance 400 according to the data migration strategy, including: the migration module 130 converts the format of each data operation request in the cache module 120 into a format supported by the target storage instance 400, and then sends each data operation request after the format conversion to the target storage instance 400.
[0078] Condition A may be that the data that has been migrated in the source storage instance 300 has reached the threshold value A. The data that has been migrated in the source storage instance 300 refers to the data that the migration module 130 in the above S105 has copied to the target storage instance 400 according to the data migration policy. Threshold A may be set by the user, or it may be dynamically set by the migration module 130 according to actual conditions. For example, threshold A may be set to the total data volume of all data in the source storage instance 300. It should be understood that when threshold A is the total data volume of all data in the source storage instance 300, it means that the migration module 130 has copied all the data in the source storage instance 300 to the target storage instance 400, so this step (i.e., S106) can be executed next.
[0079] It should also be understood that, in the process of the migration module 130 migrating the data in the source storage instance 300 to the target storage instance 400, the application 200 may initiate a data operation request to the source storage instance 300. Since the storage interface module 110 caches the data operation request to the cache module 120 while sending the data operation request to the source storage instance 300, and the source storage instance 300 is updated according to the data operation request, the migration module 130 will send the data operation request to the target storage instance 400. Therefore, if the migration module 130 copies all the data in the source storage instance 300 to the target storage instance 400 before sending the data operation request in the cache module 120 to the target storage instance 400, the target storage instance 400 may repeatedly perform multiple data operations, thereby affecting the cost and efficiency of data migration. To this end, the threshold A can also be set to a value less than the total amount of all data in the source storage instance 300. For example, the source storage instance 300 may take a snapshot before being updated according to a data operation request in the cache module 120 , and the threshold A may be set to the total amount of data in the snapshot.
[0080] S107: The migration module 130 determines whether the data operation request in the cache module 120 satisfies condition B. If yes, the following S108 is executed, and the above S106 is continued to be executed at the same time. If no, the above S106 is continued to be executed.
[0081] Condition B includes one or more of the following: the number of data operation requests in the cache module 120 is less than the threshold value B, and the amount of data in the data operation requests in the cache module 120 is less than the threshold value C. Wherein, both the threshold value B and the threshold value C can be set by the user, or can be dynamically adjusted by the migration module 130 according to the actual situation, but no matter which setting method is used, the setting of the threshold value B and the threshold value C should follow the following rules: set the threshold value B and the threshold value C as small as possible to ensure that the time for which the new data operation request from the application 200 is suspended is as short as possible. The reason is as follows: It can be seen from the following text that when the data operation request in the cache module 120 meets the condition B, the storage interface module 110 will suspend the new data operation request until the data operation request in the cache module 120 meets the condition C, and the storage interface module 110 will process the above-mentioned suspended data operation request. That is to say, the application 200 will not be able to perform data operations on the source storage instance 300 for a period of time. In order to minimize the impact of this situation on the application 200, the threshold value B and the threshold value C should be set as small as possible so that the time for which the new data operation request is suspended is very short. For the application 200, when the new data operation request is suspended for a very short time, due to the existence of the storage interface module 110, the application 200 will not perceive that the service is interrupted, thereby achieving uninterrupted application 200 during the entire data migration process.
[0082] S108: The storage interface module 110 suspends a new data operation request.
[0083] Specifically, when the migration module 130 determines that the data operation request in the cache module 120 satisfies condition B, the migration module 130 sends a notification message C to the management module 140. Accordingly, the management module 140 receives the notification message C sent by the migration module 130, and sends a notification message D to the storage interface module 110 according to the notification message C. After receiving the notification message D sent by the management module 140, the storage interface module 110 suspends the new data operation request. Here, "suspending the new data operation request" means that the storage interface module 110 stops sending new data operation requests to the source storage instance 300, and stops caching the new data operation request to the cache module 120. It should be understood that when the new data operation request is suspended, the source storage instance 300 and the target storage instance 400 will not be updated according to the new data operation request, thereby ensuring the data consistency of the source storage instance 300 and the target storage instance 400 during the data migration process.
[0084] S109: The migration module 130 determines whether the data operation request in the cache module 120 satisfies the condition C. If yes, the following S110 is executed. If no, the above S106 and S108 are continued to be executed.
[0085] Condition C includes that the number of data operation requests and / or the amount of data in the cache module 120 is 0. It should be understood that when the data operation request in the cache module 120 does not meet the condition C, it means that there are still data operation requests cached in the current cache module 120, that is, the data in the source storage instance 300 and the target storage instance 400 may be inconsistent. Therefore, in this case, the migration module 130 needs to continue to execute the above S106. On the contrary, when the data operation request in the cache module 120 meets the condition C, it means that there are no data operation requests cached in the current cache module 120, that is, the data in the source storage instance 300 and the target storage instance 400 are completely consistent. Therefore, in this case, the subsequent steps (i.e., S110) can be executed.
[0086] S110 : The storage interface module 110 stops sending new data operation requests to the source storage instance 300 , and sends new data operation requests to the target storage instance 400 .
[0087] Specifically, when the migration module 130 determines that the data operation request in the cache module 120 satisfies the condition C, the migration module 130 sends a notification message E to the management module 140. Accordingly, the management module 140 receives the notification message E sent by the migration module 130, and sends a notification message F to the storage interface module 110 according to the notification message E. After receiving the notification message F sent by the management module 140, the storage interface module 110 establishes a connection between the storage interface module 110 and the target storage instance 400. Then, the storage interface module 110 stops sending new data operation requests to the source storage instance 300, and sends new data operation requests to the target storage instance 400 according to the data migration policy.
[0088] Further, when the source storage instance 300 and the target storage instance 400 are homogeneous storage instances, the storage interface module 110 sends a new data operation request to the target storage instance 400 according to the data migration strategy, including: the storage interface module 110 sends a new data operation request to the target storage instance 400. When the source storage instance 300 and the target storage instance 400 are heterogeneous storage instances, the storage interface module 110 sends a new data operation request to the target storage instance 400 according to the data migration strategy, including: the storage interface module 110 converts the format of the new data operation request into a format supported by the target storage instance 400, and then sends the format-converted data operation request to the target storage instance 400.
[0089] Optionally, after receiving the notification message F, the storage interface module 110 may also disconnect the connection between the storage interface module 110 and the source storage instance 300 .
[0090] It should be noted that the new data operation request in this step includes the data operation request suspended in the above S108. In addition, the new data operation request in this step may also include the data operation request from the application 200 after the data migration task is completed, that is, after the data migration task is completed, the application 200 still accesses the target storage instance 400 through the storage interface module 110. In this case, the data operation request from the application 200 will be sent to the target storage instance 400 through the storage interface module 110.
[0091] S111 : the storage interface module 110 establishes a connection between the application 200 and the target storage instance 400 .
[0092] Specifically, the storage interface module 110 can establish a connection between the application 200 and the target storage instance 400 in a variety of ways. For example, the storage interface module 110 sends a notification message to the application 200, and the application 200 establishes a connection with the target storage instance 400 according to the notification message. For another example, the storage interface module 110 sends notification messages to the application 200 and the target storage instance 400 respectively, so that the application 200 establishes a connection with the target storage instance 400. It is worth noting that if the source storage instance 300 and the target storage instance 400 are heterogeneous storage instances, after the application 200 establishes a connection with the target storage instance 400, the application 200 needs to be updated accordingly, so that the application 200 can send an access request to the target storage instance 400 according to the format supported by the target storage instance 400, and at the same time, the data returned by the target storage instance 400 can also be parsed.
[0093] It should be understood that this step is an optional step. If this step is performed, the subsequent application 200 can directly access the target storage instance 400. If this step is not performed, the subsequent application 200 needs to access the target storage instance 400 through the storage interface module 110.
[0094] S112 : The migration module 130 releases the source storage instance 300 .
[0095] It should be understood that through the above S101-S109, all data in the source storage instance 300 is migrated to the target storage instance 400, so the source storage instance 300 can be released. In addition, the embodiment of the present application does not limit the execution order of the above S110-S112, that is, these three steps can be executed simultaneously or in an interchangeable order.
[0096] During the data migration process, in addition to the above data operation requests, the application 200 may also initiate a read request to the source storage instance 300. Considering that the read request will not cause changes to the source storage instance 300, while the data operation request will cause changes to the source storage instance 300, the above embodiments (i.e., S101-S112) mainly describe how the data migration system 100 processes the data operation request from the application 200 during the data migration process. For the read request from the application 200, the data migration system 100 can process it through the following steps: the storage interface module 110 receives the read request from the application 200 and sends the read request to the source storage instance 300. Correspondingly, the source storage instance 300 receives the read request sent by the storage interface module 110 and returns the corresponding data to the application 200 according to the read request. After the storage interface module 110 establishes a connection with the target storage instance 400, the storage interface module 110 can also convert the format of the read request from the application 200 into a format supported by the target storage instance, and then send the read request after the format conversion to the target storage instance 400. Correspondingly, the target storage instance 400 receives the read request after format conversion sent by the storage interface module 110, and returns corresponding data to the application 200 according to the read request.
[0097] On the basis of the above S101 - S112 , combined with two specific examples, how the data migration system 100 implements data migration in these two scenarios is described.
[0098] Example 1: Assume that the database at the back end of application 200 is MySQL, and as application 200 runs, the performance specifications of the database (referred to as MySQL1) can no longer meet business requirements, so it is necessary to migrate the data in MySQL1 to another MySQL with higher performance specifications (referred to as MySQL2). In this scenario, the data migration system 100 can implement data migration through the following steps:
[0099] The tenant logs in to the cloud management platform and enters configuration information on the management module 140 through the cloud management platform, for example, including: the name of MySQL1, the name of MySQL2, and the data migration strategy. After that, the management module 140 sends notification messages to the storage interface module 110 and the migration module 130 respectively. After receiving the notification message sent by the management module 140, the storage interface module 110 sends the data operation request from the application 200 to the source storage instance 300, and also caches it to the cache module 120. After receiving the notification message sent by the management module 140, the migration module 130 copies each data in MySQL1 to MySQL2. Then, the migration module 130 reads the data operation request from the cache module 120 and sends the data operation request to MySQL2. When the data operation request in the cache module 120 meets condition B, the migration module 130 notifies the management module 140 of the message. Accordingly, the management module 140 notifies the storage interface module 110 to suspend the new data operation request. When the data operation request in the cache module 120 meets the condition C, the migration module 130 notifies the management module 140 of the message. Accordingly, the management module 140 notifies the storage interface module 110 to establish a connection with MySQL2 and sends the above-mentioned suspended data operation request to MySQL2. After that, the storage interface module 110 establishes a connection between the application 200 and MySQL2, and disconnects the connection between the application 200 and MySQL1, so that the subsequent application 200 can directly access MySQL2. Finally, the migration module 130 releases MySQL1.
[0100] Example 2: Assume that the database of the backend of the current application 200 is MongoDB. In order to meet business needs, the data in MongoDB needs to be migrated to ElasticSearch. In this scenario, the data migration system 100 can achieve data migration through the following steps:
[0101] The tenant logs in to the cloud management platform and enters configuration information on the management module 140 through the cloud management platform, such as the name of MongoDB, the name of ElasticSearch, and the data migration strategy. After that, the management module 140 sends notification messages to the storage interface module 110 and the migration module 130 respectively. After receiving the notification message sent by the management module 140, the storage interface module 110 sends the data operation request from the application 200 to the source storage instance 300 and caches it to the cache module 120. After receiving the notification message sent by the management module 140, the migration module 130 converts the format of each data in MongoDB into a format supported by ElasticSearch, and then copies the format-converted data to ElasticSearch. Then, the migration module 130 reads the data operation request from the cache module 120, converts the format of the data operation request into a format supported by ElasticSearch, and then sends the format-converted data operation request to ElasticSearch. When the data operation request in the cache module 120 meets condition B, the migration module 130 notifies the management module 140 of the message. Accordingly, the management module 140 notifies the storage interface module 110 to suspend new data operation requests. When the data operation request in the cache module 120 satisfies condition C, the migration module 130 notifies the management module 140 of the message. Accordingly, the management module 140 notifies the storage interface module 110 to establish a connection with ElasticSearch, and converts the format of the above-mentioned suspended data operation request into a format supported by ElasticSearch, and then sends the format-converted data operation request to ElasticSearch. Afterwards, the storage interface module 110 establishes a connection between the application 200 and ElasticSearch, and disconnects the connection between the application 200 and MongoDB, so that the subsequent application 200 can directly access ElasticSearch. Finally, the migration module 130 releases MongoDB.
[0102] See next Figure 5 , Figure 5 The flowchart of another data migration method provided by the embodiment of the present application is shown. It should be noted that: Figure 5 In the described embodiment, the source storage instance 300 and the target storage instance 400 are isomorphic storage instances, but Figure 4 What is different from the described embodiment is that in this embodiment, the target storage instance 400 is obtained by splitting the source storage instance 300 (i.e., migrating part of the data in the source storage instance 300 to the target storage instance 400). For example, the target storage instance 400 can be a sub-library or sub-table of the source storage instance 300.
[0103] S201: The management module 140 receives the configuration of the tenant.
[0104] Similar to the configuration of the tenant in S101 above, the configuration of the tenant in this step may also include any one or more of the identifier of the source storage instance 300, the identifier of the target storage instance 400, and the data migration strategy. Wherein, unlike the data migration strategy in S101 above, the data migration strategy in this step is used to indicate any one or more of the following operations: migrating data that meets the migration requirements in the source storage instance 300 to the target storage instance 400, sending data operation requests that meet the migration requirements in the cache module 120 to the target storage instance 400, and sending requests that meet the migration requirements in new data operation requests to the target storage instance 400. In some embodiments, the data migration strategy includes a migration requirement or indicates a migration requirement. For example, if the data migration strategy is a consistent hashing algorithm, the data that meets the migration requirement refers to the data that needs to be migrated to the target storage instance 400 determined using the consistent hashing algorithm, and the data operation request that meets the migration requirement refers to the data operation request that needs to be sent to the target storage instance 400 determined using the consistent hashing algorithm.
[0105] S202 : the management module 140 sends a notification message A to the storage interface module 110 and a notification message B to the migration module 130 according to the tenant configuration. Accordingly, the storage interface module 110 receives the notification message A sent by the management module 140 , and the migration module 130 receives the notification message B sent by the management module 140 .
[0106] S203 : the storage interface module 110 receives a data operation request from the application 200 .
[0107] S204 : The storage interface module 110 sends the data operation request to the source storage instance 300 , and caches the data operation request to the cache module 120 .
[0108] S205 : The migration module 130 migrates the data in the source storage instance 300 to the target storage instance 400 .
[0109] Specifically, the migration module 130 obtains the identifier of the source storage instance 300 , the identifier of the target storage instance 400 , and the data migration policy according to the notification message B, and then migrates the data in the source storage instance 300 to the target storage instance 400 according to the data migration policy.
[0110] In some embodiments, the migration module 130 migrates the data in the source storage instance 300 to the target storage instance 400 according to the data migration strategy, including: the migration module 130 reads the data from the source storage instance 300, and determines whether the data meets the migration requirements according to the data migration strategy. If yes, the migration module 130 copies the data to the target storage instance 400. If not, the migration module 130 continues to read the next piece of data from the source storage instance 300, and determines whether the next piece of data meets the migration requirements according to the data migration strategy, until condition A is met (see the relevant description of S206 below).
[0111] S206 : The migration module 130 sends the data operation request in the cache module 120 to the target storage instance 400 .
[0112] Specifically, when condition A is met, the migration module 130 sends the data operation request in the cache module 120 to the target storage instance 400 according to the data migration strategy. Accordingly, the target storage instance 400 receives the data operation request sent by the migration module 130 and updates (for example, adds data, deletes data, or modifies data) according to the data operation request. It should be understood that when condition A is not met, the migration module 130 continues to execute the above S205.
[0113] Taking a data operation request in the cache module 120 as an example, the migration module 130 sends the data operation request in the cache module 120 to the target storage instance 400 according to the data migration strategy, including: the migration module 130 reads a data operation request from the cache module 120, and determines whether the data operation request meets the migration requirements according to the data migration strategy. If yes, the migration module 130 sends the data operation request to the target storage instance 400. If not, the migration module 130 continues to read the next data operation request, and determines whether the next data operation request meets the migration requirements according to the data migration strategy, until the data operation request in the cache module 120 meets the condition C (the specific description of the condition C can be found in S209 below).
[0114] Similar to the condition A in the above S106, the condition A in this step may be that the data migrated in the source storage instance 300 reaches the threshold A. The threshold A may be set by the user, or may be dynamically set by the migration module 130 according to the actual request. For example, the threshold A may be set to the total amount of data in the source storage instance 300 that meets the migration requirements, or the total amount of data in the snapshot that meets the migration requirements. The snapshot is a snapshot of the source storage instance 300 before it is updated according to the data operation request in the cache module 120.
[0115] In addition, condition A may also be that the processed data in the source storage instance 300 reaches a threshold value D. The processed data in the source storage instance 300 refers to the data that the migration module 130 has determined in S205 above whether it meets the migration requirements according to the data migration policy. Similar to threshold value A, threshold value D may be set by the user or dynamically set by the migration module 130 according to actual conditions. For example, threshold value D may be the total amount of all data in the source storage instance 300, or the total amount of data in the above snapshot.
[0116] S207: The migration module 130 determines whether the data operation request in the cache module 120 satisfies condition B. If yes, the following S208 is executed, and the above S206 is continued to be executed at the same time. If no, the above S206 is continued to be executed.
[0117] S208: The storage interface module 110 suspends a new data operation request.
[0118] S209: The migration module 130 determines whether the data operation request in the cache module 120 satisfies the condition C. If yes, the following S210 is executed. If no, the above S206 and S208 are continued to be executed.
[0119] S210 : The storage interface module 110 sends a new data operation request to the source storage instance 300 and the target storage instance 400 .
[0120] Specifically, when the migration module 130 determines that the data operation request in the cache module 120 satisfies the condition C, the migration module 130 sends a notification message E to the management module 140. Accordingly, the management module 140 receives the notification message E sent by the migration module 130, and sends a notification message F to the storage interface module 110 according to the notification message E. After receiving the notification message F sent by the management module 140, the storage interface module 110 establishes a connection between the storage interface module 110 and the target storage instance 400, and then sends a new data operation request to the source storage instance 300 and the target storage instance 400 according to the data migration policy.
[0121] Taking a new data operation request as an example, the storage interface module 110 sends the new data operation request to the source storage instance 300 and the target storage instance 400 according to the data migration policy, including: the storage interface module 110 determines whether the data operation request meets the migration requirements according to the data migration policy, and if so, the storage interface module 110 sends the data operation request to the target storage instance 400, so that the target storage instance 400 is updated according to the data operation request. If not, the storage interface module 110 sends the data operation request to the source storage instance 300, so that the source storage instance 300 is updated according to the data operation request.
[0122] It should be noted that the new data operation request in this step includes the data operation request suspended in the above S208. In addition, the new data operation request in this step may also include the data operation request from the application 200 after the data migration task is completed, that is, after the data migration task is completed, the application 200 still accesses the source storage instance 300 and the target storage instance 400 through the storage interface module 110. In this case, the data operation request from the application 200 will be sent to the source storage instance 300 and the target storage instance 400 through the storage interface module 110.
[0123] It is worth noting that, unlike the above S110, in this step, after the storage interface module 110 receives the notification message F, there is no need to execute the step of disconnecting the connection with the source storage instance 300.
[0124] S211 : the storage interface module 110 establishes a connection between the application 200 and the target storage instance 400 .
[0125] Similar to the above S111, in this step, the storage interface module 110 can also establish a connection between the application 200 and the target storage instance 400 in a variety of ways. It is worth noting that after the application 200 establishes a connection with the target storage instance 400, the application 200 needs to be updated accordingly, so that the application 200 can send a read request to the source storage instance 300 and the target storage instance 400 respectively, or can determine which storage instance the data to be queried needs to be read from, so as to send a read request to the corresponding storage instance.
[0126] It should be understood that this step is an optional step. If this step is performed, the subsequent application 200 can directly access the target storage instance 400. If this step is not performed, the subsequent application 200 needs to access the target storage instance 400 through the storage interface module 110.
[0127] S212 : The migration module 130 deletes the data in the source storage instance 300 that has been migrated to the target storage instance 400 .
[0128] It should be understood that this step can release some resources in the source storage instance 300, thereby improving the performance (e.g., read and write speed, etc.) of the source storage instance 300. In addition, the embodiment of the present application does not limit the execution order of the above S210-S212, that is, these three steps can be executed simultaneously, or they can be executed in an interchangeable order.
[0129] Similar to the above S101-S112, this embodiment (i.e., S201-S212) also mainly describes how the data migration system 100 processes the data operation request from the application 200 during the data migration process. For the read request from the application 200, the data migration system 100 can process it through the following steps: the storage interface module 110 receives the read request from the application 200 and sends the read request to the source storage instance 300. Correspondingly, the source storage instance 300 receives the read request sent by the storage interface module 110 and returns the corresponding data to the application 200 according to the read request. After the storage interface module 110 establishes a connection with the target storage instance 400, the storage interface module 110 can send the read request from the application 200 to the source storage instance 300 and the target storage instance 400 respectively, and then merge and deduplicate the results returned by the source storage instance 300 and the target storage instance 400 to obtain the target data, and finally return the target data to the application 200.
[0130] In addition, it should be noted that, for the sake of simplicity, this embodiment does not describe each step in the above S201-S212 in detail, and its specific implementation process can refer to the above S101-S112.
[0131] On the basis of the above S201 - S212 , combined with a specific example (ie, the target storage instance 400 is a sub-table of the source storage instance 300 ), how the data migration system 100 implements data migration in this scenario is described.
[0132] Assume that the amount of data in the data table (TableA) at the backend of the current application 200 is very large. In order to improve the performance of TableA, TableA needs to be split into two small tables. Among them, TableA can be split according to the consistent hash of the ID, and the two small tables obtained by the split are TableA and TableA1. TableA before the split is the source storage instance 300, and TableA1 is the target storage instance 400. In this scenario, the data migration system 100 can implement data migration through the following steps:
[0133] The tenant logs in to the cloud management platform and enters configuration on the management module 140 through the cloud management platform, including, for example, the table name of TableA, the table name of TableA1, and the data migration strategy. The data migration strategy may be as follows:
[0134] ConsistentHash consistentHash=new ConsistentHash("TableA","TableA1")
[0135] String tableName=consistentHash.hash(id)
[0136] Afterwards, the management module 140 sends notification messages to the storage interface module 110 and the migration module 130 respectively. After receiving the notification message sent by the management module 140, the storage interface module 110 sends the data operation request from the application 200 to the source storage instance 300 and caches it in the cache module 120. After receiving the notification message sent by the management module 140, the migration module 130 traverses TableA and uses the consistent hashing algorithm for each data in TableA to determine the data table to be written, thereby copying the data in TableA that meets the migration requirements to TableA1. Then, the migration module 130 reads the data operation request from the cache module 120 and uses the consistent hashing algorithm to determine whether the data operation request meets the migration requirements, thereby sending the data operation request that meets the migration requirements to TableA1. When the data operation request in the cache module 120 meets condition B, the migration module 130 notifies the management module 140 of the message. Accordingly, the management module 140 notifies the storage interface module 110 to suspend the new data operation request from the application 200. When the data operation request in the cache module 120 meets condition C, the migration module 130 notifies the management module 140 of the message. Accordingly, the management module 140 notifies the storage interface module 110 to establish a connection with TableA1, and sends the above-mentioned suspended data operation request to TableA and TableA1 according to the consistent hash. After that, the storage interface module 110 establishes a connection between the application 200 and TableA1, so that the application 200 can directly access TableA and TableA1. Finally, the migration module 130 deletes the data in TableA that has been migrated to TableA1.
[0137] The present application also provides a computing device. Figure 6 As shown, the computing device 500 includes a bus 510, a processor 520, a memory 530, and a communication interface 540. The processor 520, the memory 530, and the communication interface 540 communicate with each other through the bus 510. The computing device 500 can be a server or a terminal device. It should be understood that the embodiment of the present application does not limit the number of processors and memories in the computing device 500.
[0138] The bus 510 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 6 Although only one line is used in the figure, this does not mean that the computing device 500 has only one bus or one type of bus. The bus 510 may include a path for transmitting information between various components of the computing device 500 (eg, memory 530, processor 520, communication interface 540).
[0139] The processor 520 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0140] The memory 530 may include a volatile memory, such as a random access memory (RAM). The memory 530 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).
[0141] The memory 530 stores executable program codes. The processor 520 executes the executable program codes stored in the memory 530 to respectively implement the functions of the aforementioned storage interface module 110, cache module 120, migration module 130, and management module 140, thereby implementing the data migration method described above. In other words, the memory 530 stores instructions for executing the data migration method described above.
[0142] The communication interface 540 uses a transceiver module such as but not limited to a network interface card or a transceiver to implement communication between the computing device 500 and other devices or a communication network. For example, the computing device 500 communicates with the source storage instance 300 and the target storage instance 400 through the communication interface 540.
[0143] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes multiple computing devices. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.
[0144] like Figure 7 As shown, the computing device cluster 600 includes multiple computing devices 500. The memory 530 in one or more computing devices 500 in the computing device cluster 600 may store the same instructions for executing the data migration method described above. Alternatively, the memory 530 in one or more computing devices 500 in the computing device cluster 600 may also store partial instructions for executing the data migration method described above. That is, the combination of one or more computing devices 500 can jointly execute the instructions of the data migration method described above.
[0145] It should be noted that the memory 530 in different computing devices 500 in the computing device cluster 600 may store different instructions to respectively implement partial functions of the data migration system 100. In other words, the instructions stored in the memory 530 in different computing devices 500 may implement the functions of one or more of the storage interface module 110, cache module 120, migration module 130 and management module 140 described above.
[0146] In some embodiments, one or more computing devices 500 in the computing device cluster 600 may be connected via a network, which may be a wide area network or a local area network. Figure 8 A possible implementation is shown, such as Figure 8 As shown, the computing device cluster 600 includes four computing devices 500, namely computing device 500A, computing device 500B, computing device 500C and computing device 500D, and these four computing devices are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 530 in the computing device 500A stores instructions for executing the functions of the storage interface module 110, the memory 530 in the computing device 500B stores instructions for executing the functions of the cache module 120, the memory 530 in the computing device 500C stores instructions for executing the functions of the migration module 130, and the memory 530 in the computing device 500D stores instructions for executing the functions of the management module 140. It should be understood that Figure 8The connection mode between different computing devices 500 in the computing device cluster 600 is shown to facilitate distributed expansion and fault isolation by considering that different modules are deployed on different computing devices. In actual applications, there may be other connection modes between different computing devices 500 in the computing device cluster 600, for example, they may be adaptively deployed according to the computing capabilities of different computing devices 500 or specific application requirements.
[0147] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data migration method described above.
[0148] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media, wherein the available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state drive (SSD)), etc. The computer-readable storage medium includes instructions, wherein the instructions instruct the computing device to execute the data migration method described above.
[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.
Claims
1. A data migration method, characterized in that: The method is used in a data migration system, the system includes a migration module, a cache module and a storage interface module, and the method includes: The storage interface module receives a data operation request from an application; The storage interface module sends the data operation request to the source storage instance, and caches the data operation request to the cache module; The migration module migrates the data in the source storage instance to the target storage instance; The migration module sends the data operation request in the cache module to the target storage instance; The storage interface module stops sending new data operation requests to the source storage instance, and sends new data operation requests to the target storage instance.
2. The method according to claim 1, characterized in that The method further comprises: When the data operation request in the cache module meets the preset conditions, the storage interface module suspends the new data operation request, and the preset conditions include any one or more of the following: the number of data operation requests in the cache module is less than a first threshold, and the amount of data in the data operation request in the cache module is less than a second threshold.
3. The method according to claim 2, characterized in that The storage interface module sends the new data operation request to the target storage instance, including: When the number of data operation requests and / or the amount of data in the cache module is 0, the storage interface module sends a new data operation request to the target storage instance.
4. The method according to any one of claims 1 to 3, characterized in that: The source storage instance and the target storage instance are heterogeneous databases.
5. The method according to claim 4, characterized in that The system further includes a management module, the management module supporting receiving a tenant's configuration and supporting displaying a status of data migration to the tenant; The configuration of the tenant includes any one or more of the following: the identifier of the source storage instance, the identifier of the target storage instance, and a data migration strategy; The data migration strategy indicates any one or more of the following operations: converting the format of the data in the source storage instance into a format supported by the target storage instance, and migrating the format-converted data to the target storage instance; Converting the format of the data operation request in the cache module into a format supported by the target storage instance, and sending the data operation request after the format conversion to the target storage instance; The format of the new data operation request is converted into a format supported by the target storage instance, and the new data operation request after the format conversion is sent to the target storage instance.
6. The method according to claim 5, characterized in that The storage interface module sends the new data operation request to the target storage instance, including: The storage interface module converts the format of the new data operation request into a format supported by the target storage instance according to the data migration strategy, and sends the new data operation request after the format conversion to the target storage instance.
7. A data migration method, characterized in that: The method is used in a data migration system, the system includes a migration module, a cache module and a storage interface module, and the method includes: The storage interface module receives a data operation request from an application; The storage interface module sends the data operation request to the source storage instance, and caches the data operation request to the cache module; The migration module migrates the data in the source storage instance to the target storage instance; The migration module sends the data operation request in the cache module to the target storage instance; The storage interface module sends the new data operation request to the source storage instance and the target storage instance.
8. The method according to claim 7, characterized in that The method further comprises: When the data operation request in the cache module meets the preset conditions, the storage interface module suspends the new data operation request, and the preset conditions include any one or more of the following: the number of data operation requests in the cache module is less than a first threshold, and the amount of data in the data operation request in the cache module is less than a second threshold.
9. The method according to claim 8, characterized in that The storage interface module sends the new data operation request to the source storage instance and the target storage instance, including: When the number of data operation requests and / or the amount of data in the cache module is 0, the storage interface module sends a new data operation request to the source storage instance and the target storage instance.
10. The method according to any one of claims 7 to 9, characterized in that: The target storage instance is a sub-table of the source storage instance.
11. The method according to claim 10, characterized in that The system further includes a management module, the management module supporting receiving a tenant's configuration and supporting displaying a status of data migration to the tenant; The configuration of the tenant includes any one or more of the following: the identifier of the source storage instance, the identifier of the target storage instance, and a data migration strategy; The data migration strategy indicates any one or more of the following operations: migrating data in the source storage instance that meets the migration requirements to the target storage instance, sending data operation requests in the cache module that meet the migration requirements to the target storage instance, and sending new data operation requests that meet the migration requirements to the target storage instance.
12. A data migration system, characterized in that: include: A storage interface module, used for receiving data operation requests from applications; Sending the data operation request to a source storage instance, and caching the data operation request to a cache module; Stop sending new data operation requests to the source storage instance, and send new data operation requests to the target storage instance; A cache module, used for caching the data operation request; A migration module, used to migrate data in the source storage instance to the target storage instance; The data operation request in the cache module is sent to the target storage instance.
13. The system according to claim 12, characterized in that The storage interface module is also used to suspend new data operation requests when the data operation requests in the cache module meet preset conditions, and the preset conditions include any one or more of the following: the number of data operation requests in the cache module is less than a first threshold, and the amount of data in the data operation requests in the cache module is less than a second threshold.
14. The system according to claim 13, characterized in that The storage interface module is used to send a new data operation request to the target storage instance when the number of data operation requests and / or the amount of data in the cache module is 0.
15. The system according to any one of claims 12 to 14, characterized in that: The source storage instance and the target storage instance are heterogeneous databases.
16. The system according to claim 15, characterized in that The system also includes a management module, The management module is used to receive the tenant's configuration and display the status of data migration to the tenant; The configuration of the tenant includes any one or more of the following: the identifier of the source storage instance, the identifier of the target storage instance, and a data migration strategy; The data migration strategy indicates any one or more of the following operations: converting the format of the data in the source storage instance into a format supported by the target storage instance, and migrating the format-converted data to the target storage instance; Converting the format of the data operation request in the cache module into a format supported by the target storage instance, and sending the data operation request after the format conversion to the target storage instance; The format of the new data operation request is converted into a format supported by the target storage instance, and the new data operation request after the format conversion is sent to the target storage instance.
17. The system according to claim 16, characterized in that The storage interface module is used to convert the format of the new data operation request into a format supported by the target storage instance according to the data migration strategy, and send the new data operation request after the format conversion to the target storage instance.
18. A data migration system, characterized in that: include: A storage interface module, used for receiving data operation requests from applications; Sending the data operation request to a source storage instance, and caching the data operation request to a cache module; Sending the new data operation request to the source storage instance and the target storage instance; A cache module, used for caching the data operation request; A migration module, used to migrate data in the source storage instance to the target storage instance; The data operation request in the cache module is sent to the target storage instance.
19. The system according to claim 18, characterized in that The storage interface module is also used to suspend new data operation requests when the data operation requests in the cache module meet preset conditions, and the preset conditions include any one or more of the following: the number of data operation requests in the cache module is less than a first threshold, and the amount of data in the data operation requests in the cache module is less than a second threshold.
20. The system according to claim 19, characterized in that The storage interface module is used to send a new data operation request to the source storage instance and the target storage instance when the number of data operation requests and / or the amount of data in the cache module is 0.
21. The system according to any one of claims 18 to 20, characterized in that The target storage instance is a sub-table of the source storage instance.
22. The system according to claim 21, characterized in that The system also includes a management module, The management module is used to receive the tenant's configuration and display the status of data migration to the tenant; The configuration of the tenant includes any one or more of the following: the identifier of the source storage instance, the identifier of the target storage instance, and a data migration strategy; The data migration strategy indicates any one or more of the following operations: migrating data in the source storage instance that meets the migration requirements to the target storage instance, sending data operation requests in the cache module that meet the migration requirements to the target storage instance, and sending new data operation requests that meet the migration requirements to the target storage instance.
23. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in a memory of the at least one computing device to implement the method according to any one of claims 1 to 11.
24. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 11.
25. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, which, when executed by a computing device cluster, cause the computing device cluster to perform the method according to any one of claims 1 to 11.
