A data migration method and device

Abstract

The application discloses a data migration method and device, the method comprises the following steps: receiving migration instruction information; the migration instruction information at least comprises an established subtable, an original table storing data, and a migration data volume at each historical data migration time; based on the established subtable, an incremental data migration task is established, migration logic corresponding to the incremental data migration task is determined, and based on the migration logic, the incremental data migration task is executed to obtain a first subtable comprising incremental data; based on the data stored in the original table, the migration data volume at each historical data migration time, and a preset creation rule, at least one historical data migration task is created, and based on a preset timing scheduling task, the at least one historical data migration task is executed to obtain a second subtable comprising historical data to be migrated; and based on the first subtable and the second subtable, a target subtable is obtained. The above method realizes data migration when an application is running online without stopping, and improves the reliability of the application during the data migration process.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of financial technology (Fintech), and more particularly to a data migration method and apparatus. Background Technology

[0002] With the development of computer technology, more and more technologies are being applied in the financial field. The traditional financial industry is gradually transforming into financial technology. However, due to the security and real-time requirements of the financial industry, higher demands are being placed on technology.

[0003] Currently, with the rapid development of cloud computing and big data, internet services have brought great convenience to people's lives. However, with the continuous expansion of various businesses, the amount of data that needs to be stored is also increasing. In related technologies, data storage typically uses a single database and single table approach. However, when a single table has tens of millions or more records, the read and write performance of the corresponding table will drop sharply, leading to timeouts in service response times. Therefore, database capacity planning is necessary. For example, the number of records in a single table should not exceed ten million. If the data exceeds ten million records and the data in the corresponding tables continues to increase, to ensure that service performance is not degraded, it is necessary to shard the database and reduce the data capacity of a single database and single table; that is, data migration is required.

[0004] However, in related technologies, data migration is usually carried out after the application corresponding to the business has stopped providing services to the outside world. After the data migration, the service of the application corresponding to the business is then brought back online. This will lead to business losses and reduce the user experience. Summary of the Invention

[0005] This invention provides a data migration method and apparatus to solve the problem in the prior art that data migration requires service interruption, and that abnormal situations are prone to occur after service is restored, leading to reduced application reliability.

[0006] In a first aspect, the present invention provides a data migration method, comprising: receiving migration instruction information; wherein the migration instruction information includes at least an established partitioned table, an original table storing data, and the amount of migration data for each historical data migration; based on the established partitioned table, establishing an incremental data migration task, determining the migration logic corresponding to the incremental data migration task, and executing the incremental data migration task based on the migration logic to obtain a first partitioned table including incremental data; the migration logic is used to determine the incremental data between the original table and the partitioned table, and to synchronize the incremental data; based on the data stored in the original table, the amount of migration data for each historical data migration, and a preset creation rule, creating at least one historical data migration task, executing the at least one historical data migration task to obtain a second partitioned table including historical data to be migrated; and combining the first partitioned table and the second partitioned table to obtain a corresponding target partitioned table including incremental data and historical data to be migrated.

[0007] Based on the above method, the incremental data to be synchronized in the original table can be accurately identified based on the migration logic, and an incremental data migration task can be created. Furthermore, historical data to be migrated can be identified, and historical data migration tasks can be created. This allows data that needs to be synchronized in the original table to be migrated to the established partitioned table. This avoids conflicts arising from simultaneously synchronizing incremental and historical data, thus ensuring continuous online service during data migration without interrupting external services.

[0008] Optionally, the migration logic includes at least: a data coloring sub-logic and an open database modification log sub-logic; the coloring sub-logic is used to determine whether the data stored in the original table has been modified after being migrated to the partitioned table; the open database modification log sub-logic is used to record the addition and deletion information of each piece of data.

[0009] Based on the migration logic described above, the incremental data between the original table and the partitioned table can be accurately and quickly determined.

[0010] Optionally, based on the migration logic, the incremental data migration task is executed to obtain a first partition table including incremental data, including: adding two corresponding data coloring fields to both the original table and the partition table; determining the incremental data that is different between the original table and the partition table based on the two corresponding data coloring fields and the data coloring sub-logic in the migration logic, and migrating the incremental data to the corresponding partition table to obtain an initial first partition table; setting the original table and the initial first partition table to open the MySQL binary log using row format, and listening to the MySQL binary log to extract the data addition and deletion information corresponding to the original table and the initial first partition table; the MySQL binary log includes each operation performed on the original table and the initial first partition table, as well as the data added, deleted, and updated based on the operation; translating the data addition and deletion information to obtain the data corresponding to the data addition and deletion information, and determining the data corresponding to the data addition and deletion information as the data included in the corresponding first partition table to obtain the first partition table.

[0011] Based on the data coloring logic provided by the above method, different incremental data in the original table and the partitioned table can be automatically and quickly identified to obtain the first partitioned table.

[0012] Optionally, based on the data coloring fields and the data coloring sub-logic in the migration logic, the incremental data that differs between the original table and the partitioned table is determined, including: if it is determined that the first data coloring field and the second data coloring field of the first data in the MySQL binary log corresponding to the partitioned table are consistent and have a preset format, then the first data is determined to be an incremental data; or, if it is determined that the timestamp corresponding to the first data coloring field of the second data in the original table is later than the timestamp corresponding to the second data coloring field of the second data, then the second data is determined to be an incremental data synchronized from the partitioned table to the original table; or, if it is determined that the third data in the original table has not been modified, and the first data coloring field corresponding to the third data is consistent with the second data coloring field corresponding to the third data and does not have the preset format, then the third data is determined to be a duplicate incremental data.

[0013] Based on the data coloring field in the above method, different incremental data in the original table and the partitioned table can be automatically and quickly identified.

[0014] Optionally, based on the data stored in the original table, the amount of migration data during each historical data migration, and preset creation rules, a historical data migration task is created to obtain at least one historical data migration task. This includes: obtaining the smallest data identifier in the data stored in the original table and assigning the smallest data identifier to an initial migration start variable; using the amount of migration data during each historical data migration as a loop variable for cyclic migration data; based on the loop variable, obtaining from the original table the last data record of a single cycle migration determined by the amount of migration data; when two data records are obtained, determining the migration start variable and migration end variable for the next cycle based on the two data records and the amount of migration data; creating a corresponding historical data migration task based on the migration start variable and migration end variable; wherein the migration start variable is used to represent the data identifier corresponding to the first data record in each sub-table; the migration end variable is used to represent the data identifier corresponding to the last data record in each sub-table; if no two data records are obtained, it is determined that the historical data migration task has been created, and at least one historical data migration task is obtained.

[0015] Based on the above method, historical data migration tasks can be created simply and efficiently using preset creation rules, providing a good foundation for subsequent data migration and improving the efficiency of data migration.

[0016] Optionally, executing the at least one historical data migration task to obtain a second table including historical data to be migrated includes: when it is determined that the start time of a preset timed scheduling task has been reached, obtaining migration status information corresponding to any of the historical data migration tasks; when it is determined that the obtained migration status information is in a state to be migrated, determining the migration start data and migration end data corresponding to any of the historical data migration tasks, and locking the data between the migration start data and the migration end data based on a shared lock to obtain any of the historical data migration tasks with unmodifiable data; performing processing on any of the historical data migration tasks with unmodifiable data, and releasing the shared lock set on any of the historical data migration tasks with unmodifiable data after the processing is completed; when it is determined that the migration status information corresponding to the at least one historical data migration task is empty, determining that the historical data to be migrated has been migrated and obtaining a second table including historical data to be migrated.

[0017] Based on the above method, the scheduling time for historical data to be migrated can be customized, and data migration can be stopped at any time and resumed after interruption. This means that if the data migration puts too much pressure on the database, it can be paused at any time. Therefore, the data migration time can be determined based on actual needs, ensuring efficient data migration while maintaining uninterrupted online services.

[0018] Optionally, the processing of any of the historical data migration tasks of the unmodifiable data includes: constructing multiple migration SQL statements based on each data identifier and a preset strategy in any of the historical data migration tasks of the unmodifiable data; wherein the preset strategy is to wait for the shared lock in the original table to end before performing migration on any of the historical data migration tasks of the unmodifiable data; and setting an exclusive lock between the migration start data and migration end data of any of the historical data migration tasks of the unmodifiable data; and executing any of the historical data migration tasks of the unmodifiable data based on the multiple migration SQL statements.

[0019] Based on the above method, modifications to historical data to be migrated can be prevented during the data migration process. Furthermore, because the migration SQL statement executes very quickly, concurrency issues with the original table can be avoided, thus preventing any impact on online service performance.

[0020] Optionally, after obtaining the target table, the method further includes: sending the target table to the corresponding device, so that the device can select to use the target table or the original table based on the selection information.

[0021] Based on the above method, the corresponding device can automatically switch between using the partitioned table or the original table according to actual usage needs. In this way, if the service provided by the partitioned table fails, it can switch back to the data access layer service provided by the original table without loss, ensuring business stability.

[0022] Secondly, the present invention provides a data migration apparatus, comprising: a receiving unit for receiving migration instruction information; wherein the migration instruction information includes at least an established partitioned table, an original table storing data, and the amount of migration data for each historical data migration; a first processing unit for establishing an incremental data migration task based on the established partitioned table, determining the migration logic corresponding to the incremental data migration task, and executing the incremental data migration task based on the migration logic to obtain a first partitioned table including incremental data; the migration logic is used to determine the incremental data between the original table and the partitioned table, and to synchronize the incremental data; a second processing unit for creating at least one historical data migration task based on the data stored in the original table, the amount of migration data for each historical data migration, and a preset creation rule, executing the at least one historical data migration task to obtain a second partitioned table including historical data to be migrated; and an obtaining unit for combining the first partitioned table and the second partitioned table to obtain a corresponding target partitioned table including incremental data and historical data to be migrated.

[0023] Optionally, the migration logic includes at least: a data coloring sub-logic and an open database modification log sub-logic; the coloring sub-logic is used to determine whether the data stored in the original table has been modified after being migrated to the partitioned table; the open database modification log sub-logic is used to record the addition and deletion information of each piece of data.

[0024] Optionally, the first processing unit is specifically configured to: add two corresponding data coloring fields to both the original table and the partitioned table; based on the two corresponding data coloring fields and the data coloring sub-logic in the migration logic, determine the incremental data that is different between the original table and the partitioned table, and migrate the incremental data to the corresponding partitioned table to obtain an initial first partitioned table; set the original table and the initial first partitioned table to open the MySQL binary log using row format, and listen to the MySQL binary log to extract the data addition and deletion information corresponding to the original table and the initial first partitioned table; the MySQL binary log includes each operation performed on the original table and the initial first partitioned table, as well as the data added, deleted, and updated based on the operation; translate the data addition and deletion information to obtain the data corresponding to the data addition and deletion information, and determine the data corresponding to the data addition and deletion information as the data included in the corresponding first partitioned table to obtain the first partitioned table.

[0025] Optionally, the first processing unit is specifically configured to: determine the first data as an incremental data if it is determined that the first data coloring field and the second data coloring field of the first data in the MySQL binary log corresponding to the partitioned table are consistent and are in a preset format; or, determine the second data as an incremental data if it is determined that the timestamp corresponding to the first data coloring field of the second data in the original table is later than the timestamp corresponding to the second data coloring field of the second data; or, determine the third data as a duplicate incremental data if it is determined that the third data in the original table has not been modified, and the first data coloring field corresponding to the third data is consistent with the second data coloring field corresponding to the third data and is not in the preset format.

[0026] Optionally, the second processing unit is specifically used for: obtaining the smallest data identifier in the data stored in the original table, and assigning the smallest data identifier to the initial migration start variable; using the migration data volume during each historical data migration as the loop variable for the cyclic migration data, and based on the loop variable, obtaining the last data record of a single cyclic migration and the next data record of the last data record from the original table, determined by the migration data volume as the dividing criterion; when two data records are obtained, determining the migration start variable and migration end variable for the next cycle based on the two data records and the migration data volume; creating a corresponding historical data migration task based on the migration start variable and the migration end variable; wherein, the migration start variable is used to characterize the data identifier corresponding to the first data record in each sub-table; the migration end variable is used to characterize the data identifier corresponding to the last data record in each sub-table; if no two data records are obtained, it is determined that the historical data migration task has been created, and at least one historical data migration task is obtained.

[0027] Optionally, the second processing unit is specifically configured to: when it is determined that the start time of the preset timed scheduling task has been reached, obtain migration status information corresponding to any of the historical data migration tasks; when it is determined that the obtained migration status information is in a state to be migrated, determine the migration start data and migration end data corresponding to any of the historical data migration tasks, and lock the data between the migration start data and the migration end data based on a shared lock to obtain any of the historical data migration tasks with unmodifiable data; perform processing on any of the historical data migration tasks with unmodifiable data, and when the processing is completed, release the shared lock set on any of the historical data migration tasks with unmodifiable data; when it is determined that the state information to be migrated corresponding to at least one historical data migration task is empty, determine that the historical data to be migrated has been migrated, and obtain a second sub-table including the historical data to be migrated.

[0028] Optionally, the second processing unit is specifically configured to: construct multiple migration SQL statements based on each data identifier and a preset strategy in any of the historical data migration tasks of the unmodifiable data; wherein the preset strategy is to wait for the shared lock in the original table to end before performing migration on any of the historical data migration tasks of the unmodifiable data; and to set an exclusive lock between the migration start data and the migration end data of any of the historical data migration tasks of the unmodifiable data; and to execute any of the historical data migration tasks of the unmodifiable data based on the multiple migration SQL statements.

[0029] Optionally, the apparatus further includes a sending unit, configured to: send the target sub-table to a corresponding device, so that the device selects to use the target sub-table or the original table based on selection information.

[0030] The beneficial effects of the second aspect and the various optional devices described above can be referred to the beneficial effects of the first aspect and the various optional methods described above, and will not be repeated here.

[0031] Thirdly, the present invention provides a computer device including a program or instructions, which, when executed, are used to perform the first aspect and the various optional methods of the first aspect.

[0032] Fourthly, the present invention provides a storage medium including a program or instructions, which, when executed, perform the first aspect and the various optional methods of the first aspect. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0034] Figure 1 This is a schematic diagram of data migration in existing technologies;

[0035] Figure 2 This is a schematic diagram of an application scenario provided by an embodiment of the present invention;

[0036] Figure 3 An architecture diagram for data migration provided in an embodiment of the present invention;

[0037] Figure 4 This is a flowchart illustrating the steps of a data migration method provided in an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram of a data migration device provided in an embodiment of the present invention. Detailed Implementation

[0039] To better understand the above technical solutions, the following will describe the above technical solutions in detail with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solutions of the present invention, rather than limitations on the technical solutions of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0040] It should be noted that the terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of images can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0041] To facilitate understanding of the technical solutions provided in the embodiments of the present invention, some key terms used in the embodiments of the present invention will be explained here first:

[0042] 1. MySQL Database: MySQL is a relational database management system and one of the most popular relational database management systems currently available. Specifically, MySQL stores data in different tables rather than storing all data in one large repository, thus increasing the speed of data retrieval and storage. Furthermore, the SQL language used by MySQL is the most commonly used standardized language for accessing databases.

[0043] 2. Uninterrupted Online Service: Uninterrupted online service can be understood as the software or system providing 24 / 7 uninterrupted high availability service.

[0044] 3. Table Structure Splitting: Table structure splitting includes horizontal splitting and vertical splitting. Horizontal splitting refers to splitting the data table rows, splitting the data of a single table into multiple tables with the same structure for storage, and each split table is divided into a certain range of data based on the primary key. Vertical splitting refers to splitting the data table columns, splitting a table with many columns into multiple tables.

[0045] 4. MySQL Binary Log: The MySQL binary log (Binlog) records all data changes made in MySQL. Specifically, the row-formatted Binlog log records details of each row modification, including: information for each column in INSERT rows; records of the values ​​before and after the update for each column in UPDATE statements; and information for each column in rows deleted by DELETE statements.

[0046] 5. Shared Locks: Shared locks, also known as read locks or S locks, allow transactions T to read data object A but not modify it. Other transactions can only acquire S locks on A, not X locks, until T releases the S lock on A. This ensures that other transactions can read A, but cannot modify A before T releases the S lock on it.

[0047] 6. Exclusive Lock: An exclusive lock is also called a write lock or an X lock. If transaction T acquires an X lock on data object A, transaction T can read and modify A, while other transactions cannot acquire any locks on A until T releases the lock on A. This ensures that other transactions cannot read or modify A before T releases the lock on A.

[0048] 7. Shard: The original table (Pre-Shard Table) is split into multiple shard tables Shard_1, Shard_2...Shard_n using a sharding scheme, and the total data of all shard tables is consistent with the data of the original table.

[0049] 8. PT Tool: Percona Toolkit, or PT tool for short, is a tool for managing MySQL, including functions such as archiving and cleaning records in tables, online DDL, checking data consistency in master-slave replication, checking duplicate indexes, and locating table files with high IO usage.

[0050] The design concept of the embodiments of the present invention is briefly introduced below:

[0051] Currently, in certain systems within the internet industry, such as order systems or user information systems, the amount of data stored in these systems increases dramatically as business expands. If this order or user information is stored in a single MySQL database with a single table, the read and write performance of the table will plummet when the number of records reaches tens of millions or more, leading to timeouts for order or user-related services. Therefore, when the data in a single MySQL table exceeds a preset threshold and the data in that table continues to grow, it is necessary to shard the database and reduce the data capacity of each single database and table to ensure that service performance is not compromised.

[0052] For example, see Figure 1 The diagram illustrates table partitioning during data migration in existing technologies. Assume an e-commerce company processes over 100 million orders daily, and this number continues to grow. Order data is stored in an order table (T_ORDER), which specifically includes at least the following example fields: order number (order_id), user (user_id), and order amount (amount). When it's determined that the order data is still expanding and the order table's data volume has reached a preset threshold, requiring table partitioning and data migration, the T_ORDER table can be split according to the user field (user_id).

[0053] Specifically, the `user_id` field can be hashed and then modulo 32. This allows the data to be evenly split into 32 smaller tables: T_ORDER_1, T_ORDER_2, ..., T_ORDER_32. Then, data migration and application modifications can be performed. This ensures controllability of data records in the order sub-tables and guarantees continued reliability of the order service.

[0054] However, implementing the aforementioned solution requires stopping the order service program. This is to prevent data inflow during the migration of data from the T_ORDER table, which could cause inconsistencies between the original table and the partitioned table. Therefore, in existing technologies, large table splitting requires a limited shutdown of the service program before production deployment, which can lead to business losses. Furthermore, the PT tool is needed to import the original table data into T_ORDER_1 through T_ORDER_32 according to the user_id splitting rule. This process cannot be paused at will. If the database instance serves multiple applications simultaneously, the migration process will increase the pressure on the database central processing unit (CPU) and input / output interfaces, impacting other applications.

[0055] Therefore, embodiments of the present invention provide a data migration method that has virtually no impact on the performance of online services, allowing for continuous online service provision without interruption of external services. Furthermore, during the migration of partitioned table data, if the data migration puts excessive pressure on the database, the migration can be paused at any time.

[0056] After introducing the design concept of the embodiments of the present invention, the following is a brief introduction to the application scenarios applicable to the data migration technology solutions in the embodiments of the present invention. It should be noted that the application scenarios described in the embodiments of the present invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. As those skilled in the art will know, with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0057] In the embodiments of the present invention, please refer to Figure 2 The illustrated application scenario includes an electronic device 201 and a computer device 202. The electronic device 201 can communicate with the computer device 202, for example, by direct or indirect connection through wired or wireless communication. This invention does not impose any limitations.

[0058] In this scenario, administrators can provide services to users based on applications deployed on electronic devices 201. Electronic devices 201 store all data related to user interactions with the applications, allowing them to determine whether data migration is necessary based on the actual implementation. When electronic devices 201 determine that data migration is required, they can interact with computer devices 202 equipped with a data migration platform to complete the migration.

[0059] The computer device 202 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The electronic device 201 can be a smartphone, tablet, laptop, desktop computer, smart TV, smart wearable device, or a server, but is not limited to these.

[0060] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating a data migration method according to an embodiment of the present invention. Specifically, Figure 3 The data migration platform is deployed on a computer device, and this data migration platform is used to execute the data migration method provided by this invention. Figure 3 The order service in the order system can be understood as a service provided by a specific application in an electronic device.

[0061] In this embodiment of the invention, the electronic device corresponding to the order service of the order system can determine whether to initiate data migration using table partitioning based on the data of the current order system. Specifically, it can determine whether the data of the order system has reached a preset threshold. If the preset threshold is reached, it can determine to initiate data migration using table partitioning. After determining to use table partitioning for data migration, the electronic device can determine whether to use a horizontal partitioning scheme or a vertical partitioning scheme based on actual needs. It should be noted that the following description uses horizontal partitioning as an example.

[0062] In this embodiment of the invention, after the electronic device determines to adopt a horizontal partitioning scheme, it is also necessary to evaluate and determine the size of this migration batch, i.e., the total data storage of each partition table, as well as the start and end times of the migration. Furthermore, partition tables can be created to obtain the created partition tables.

[0063] In this embodiment of the invention, the application used by the order system can add configuration parameters to form corresponding switch function keys, and the switch function keys are used to control whether the database tables of the corresponding order service use the original tables or use tables based on a series of partitioned tables.

[0064] For example, when the switch function key is in the on state, it can be understood that before the migration of the data to be migrated from the original table to the partition table is confirmed to be correct, or before the failure of the migration of the data to be migrated from the original table to the partition table is recovered, the data access layer logic based on the original table is used; when the switch function key is in the off state, after the migration of the data to be migrated from the original table to the partition table is completed, the data access layer logic based on the new partition table is used.

[0065] In this embodiment of the invention, after the electronic device has completed the data migration preparation, it can send migration instruction information to the data migration platform. The data migration platform can then perform incremental data synchronization of the original table and the partitioned table and synchronization of historical data to be migrated based on the migration instruction information, thereby realizing data migration.

[0066] In this embodiment of the invention, when the electronic device determines that the historical data to be migrated has been migrated, it can send a partitioned table after the data migration is completed. The electronic device can then verify the data in the partitioned table. If the verification is successful, the electronic device can use a configured switch function key to switch from the original table service to the new table. Furthermore, if an anomaly is detected, the electronic device can use a configured switch to switch back to the original table, which will then provide data access capabilities.

[0067] Furthermore, once the electronic device determines that it is providing services based on the original table or the partitioned table, it can send a migration task completion instruction to the computer device. In this way, the data migration platform deployed on the computer device can delete the data coloring fields added to the original table and clean up the information corresponding to the historical migration data tasks, thus completing this data migration.

[0068] It should be noted that, in this embodiment of the invention, each established partition table can be understood as a partition table that includes field names identical to those in the original table but does not include data. The number of established partition tables can be determined based on the partitioning rules used to split the original table, and there is no specific limit to the number. Furthermore, each target partition table can be understood as a partition table based on data from a first partition table including incremental data and a second partition table including historical migration data. The partition table containing data corresponding to the established partition table is its corresponding target partition table, and the target partition table can form the corresponding original table for storing data.

[0069] To further illustrate the data migration method provided in the embodiments of the present invention, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of the present invention provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of the present invention. In actual processing or when the device executes the method, it may be executed in the order shown in the embodiments or drawings, or in parallel (e.g., in an application environment with parallel processors or multi-threaded processing).

[0070] The following combination Figure 4 The flowchart shown illustrates the data migration method in this embodiment of the invention. The method flow of this embodiment of the invention is described below.

[0071] Step 401: Receive migration instruction information; wherein the migration instruction information includes at least the established partitioned table, the original table storing the data, and the amount of migration data for each historical data migration.

[0072] In this embodiment of the invention, the data migration platform can receive migration instruction information sent by an electronic device with an application deployed thereon. The migration instruction information includes at least the established partitioned table, the original table storing the data, and the amount of migration data for each historical data migration.

[0073] Step 402: Based on the established partitioned table, establish an incremental data migration task, determine the migration logic corresponding to the incremental data migration task, and execute the incremental data migration task based on the migration logic to obtain the first partitioned table including incremental data; the migration logic is used to determine the incremental data between the original table and the partitioned table, and to synchronize the incremental data.

[0074] In this embodiment of the invention, incremental data between the original table and the partitioned table can be synchronized based on migration logic. Specifically, the migration logic includes at least: data coloring sub-logic and database modification log opening sub-logic; the coloring sub-logic is used to determine whether the data stored in the original table has been modified after being migrated to the partitioned table; the database modification log opening sub-logic is used to record the addition and deletion information of each piece of data.

[0075] In embodiments of the present invention, the first partition table including data increments can be obtained by means of, but is not limited to, the following steps:

[0076] Step A: Add two corresponding data coloring fields to both the original table and the partitioned table.

[0077] In this embodiment of the invention, the data migration platform adds fields for data coloring determination to both the original table and the partitioned table based on migration instruction information; that is, two data coloring fields. Specifically, these two data coloring fields can be represented as m_t and u_t. Here, m_t represents the timestamp of data being synchronized from the original table to the partitioned table via the data migration platform, or from the partitioned table to the original table via the data migration platform, and u_t represents the timestamp of the last update when the data record was automatically updated.

[0078] As can be seen, the relationship between the data coloring fields u_t and m_t can be used to determine whether a specific piece of data marked by the data coloring field has been modified by the application after migration.

[0079] Step B: Based on the two corresponding data coloring fields and the data coloring sub-logic in the migration logic, determine the incremental data that is different between the original table and the partition table, and migrate the incremental data to the corresponding partition table to obtain the initial first partition table.

[0080] Optionally, if it is determined that the first data coloring field and the second data coloring field of the first data in the MySQL binary log corresponding to the partitioned table are the same and are in the preset format, the first data is determined to be an incremental data.

[0081] For example, if the data migration platform receives MySQL binary log data record 1 where u_t and m_t are both '0000-00-00 00:00:00', it means that the data corresponding to data record 1 was added by the application and needs to be synchronized. Then, the data corresponding to data record 1 can be identified as an incremental data.

[0082] Optionally, if the timestamp corresponding to the first data coloring field of the second data in the original table is later than the timestamp corresponding to the second data coloring field of the second data, the second data can be determined to be an incremental data.

[0083] Specifically, the `u_t` field is automatically synchronized to the timestamp of the data update when a data record is modified. If the value of the `u_t` field is greater than the value of the `m_t` field, it means that this part of the data has been updated by the application after the data is synchronized to the target table. Therefore, in order to maintain the consistency between the original table and the target table, it is necessary to synchronize and update the original table.

[0084] Optionally, if it is determined that the third data in the original table has not been modified, and the first data coloring field corresponding to the third data is consistent with the second data coloring field corresponding to the third data and is not in a preset format, then the third data is determined as a duplicate incremental data.

[0085] Specifically, to prevent circular synchronization, after the data stored in the original table is synchronized to the sharded table, the sharded table will also generate a corresponding MySQL binary log (binlog). If the data stored in the original table has not been modified at this time, there will be cases where the m_t field and u_t field are the same but not '0000-00-00 00:00:00'. In this case, it can be determined that the data stored in the original table is a duplicate incremental data. Therefore, the data migration platform can directly ignore the data corresponding to this situation and does not process it.

[0086] Step C: Set the original table and the initial first partition table to open the MySQL binary log using row format, and listen to the MySQL binary log to capture the data addition and deletion information corresponding to the original table and the initial first partition table; the MySQL binary log includes each operation performed on the original table and the initial first partition table, as well as the data added, deleted, and updated based on the operation;

[0087] Step D: Translate the data addition / deletion information, obtain the data corresponding to the data addition / deletion information, and determine the data corresponding to the data addition / deletion information as the data included in the corresponding first partition table, so as to obtain the first partition table.

[0088] In this embodiment of the invention, the data migration platform consumes the new binlog logs generated from the original table and the partitioned table using the component mysql-binlog-connector. Specifically, each DML operation in the original and new tables, along with the corresponding data modifications, is synchronized to the binlog logs. Therefore, by listening to INSERT / UPDATE / DELETE events in the binlog, modification information (i.e., data addition and deletion) of the corresponding table data can be obtained. These modifications can then be intercepted and translated into corresponding modification operations for the partitioned table data, thereby enabling incremental data migration.

[0089] Specifically, when an application performs insert, delete, or update operations on the original table, as well as a delete operation generated by a delete statement synchronized from a partitioned table, the MySQL binary log will record these operations. Furthermore, when the MySQL binary log is synchronized to the data migration platform, the application can parse it to obtain the corresponding data insert and delete information.

[0090] In this embodiment of the invention, the data migration platform can evaluate each MySQL binary log (binlog) entry to migrate the corresponding data to the appropriate shard. Specifically, the data migration platform can determine the data to be migrated from the original table based on data coloring fields; then, according to the sharding rules, it migrates the data to be migrated from the original table to the corresponding shard.

[0091] For example, the sharding rule can be hash(user_id)%32, which can be used to calculate and determine the table partitioning.

[0092] In this embodiment of the invention, to prevent errors caused by the simultaneous migration of historical and incremental data, a `replace` statement can be used to replace `insert` during incremental data synchronization migration, and the data coloring fields `u_t` and `m_t` are set to `now()`. Furthermore, a `Replace` insert update command can be assembled based on the updated column values ​​in the database update log generated by the `update` statement, and the values ​​of all columns before the update recorded in the database update log generated by the `update` statement. That is, the latest complete incremental data can be obtained based on the modified and unmodified values ​​in the database update log. Then, a `delete ignore` statement can be assembled based on the primary key information in the `delete` statement, and data processing can be performed based on the `delete ignore` statement.

[0093] For example, the binlog receives an insert record with data identifier id 10001, but simultaneously, historical data migration also migrates this same data with data identifier id 100001. If the incremental data is also inserted, a primary key conflict will occur. Therefore, consider using a replace statement, so that the content in the binlog can overwrite the record in the partitioned table, ensuring that the data in the final partitioned table is correct. Furthermore, if the binlog updates the record with data identifier id 10001, but historical data has not yet synchronized this record, using an update statement will result in 0 updated records. However, using a replace statement allows data to be inserted into the partitioned table via the binlog while updating the original table, without waiting for historical data synchronization to migrate this data update, thus enabling faster synchronization of incremental data.

[0094] For example, the SQL statements to prevent errors caused by the simultaneous migration of historical and incremental data can be: REPLACE INTO T_ORDER_1(order_id,user_id,amount…m_t,u_t)VALUES('10086','0001',999…now(),now()); REPLACE INTO T_ORDER_1(order_id,user_id,amount…m_t,u_t)VALUES('10086','0001',10087…now(),now()); DELETE IGNORE FROM T_ORDER_1WHERE order_id='10086'.

[0095] Step 403: Based on the data stored in the original table, the amount of data migrated each time historical data migration occurs, and the preset creation rules, create at least one historical data migration task, execute at least one historical data migration task, and obtain a second partition table including historical data to be migrated.

[0096] In this embodiment of the invention, the data migration platform can obtain the smallest data identifier in the data stored in the original table and assign the smallest data identifier to the initial migration start variable.

[0097] For example, assuming the smallest data identifier is represented as order_id, the smallest data identifier in the data stored in the original table can be represented as: SELECT order_id FROM T_ORDER FORCE INDEX(`PRIMARY`) ORDER BY order_id LIMIT 1.

[0098] In this embodiment of the invention, after assigning the smallest data identifier to the initial migration start variable, the amount of migration data during each historical data migration can be used as the loop variable for the cyclic migration data. Based on the loop variable, the last data and the next data of the last data in a single cyclic migration, determined by the amount of migration data, are obtained from the original table.

[0099] Specifically, #trunk_size is used to represent the amount of data migrated each time historical data is migrated. Then, the data record of the last data record and the data record of the next data record of the last data record can be represented as: SELECT / *! 40001SQL_NO_CACHE * / order_id FROM T_ORDER FORCE INDEX(`PRIMARY`)WHERE order_id>=#batch_start_key ORDER BY order_id asc LIMIT#trunk_size,2.

[0100] For example, assuming #trunk_size is 2000, since the initial count starts from 1, the last data record returned is the data with data identifier 2001, and the next data record after the last data record is the data with data identifier 2002.

[0101] In this embodiment of the invention, when the data migration platform acquires two data records, it determines the migration start variable and migration end variable for the next cycle based on the two data records and the amount of migration data in each historical data migration. Furthermore, a corresponding historical data migration task can be created based on the migration start variable and migration end variable; wherein, the migration start variable is used to characterize the data identifier corresponding to the first data record in each sub-table; and the migration end variable is used to characterize the data identifier corresponding to the last data record in each sub-table.

[0102] Specifically, assuming the table storing the historical data migration task is named `migrate_task`, the historical data migration task can be created in the following way:

[0103] The SQL statement inserts an `INSERT INTO migrate_task(table_name, start_key, end_key, status)` statement with the values ​​'T_ORDER', '#batch_start_key', '#batch_end_key', and 'PENDING'. Here, 'table_name' represents the column name of the `migrate_task` corresponding to the historical data migration task; 'start_key' represents the data identifier of the first record in the `migrate_task`; 'end_key' represents the data identifier of the last record in the `migrate_task`; 'status' represents the migration status information; 'T_ORDER' represents the column name of the original table; '#batch_start_key' represents the migration start variable for retrieving data from the original table; '#batch_end_key' represents the migration end variable for retrieving data from the original table; and 'PENDING' represents the migration status information. It should be noted that '#batch_start_key' is determined based on the data record following the last record determined in the previous loop. Furthermore, the value of the 'batch_id' field can be determined using an auto-increment algorithm.

[0104] In this embodiment of the invention, if the data migration platform does not obtain two data records, it determines that the historical data migration task has been created and at least one historical data migration task has been obtained. Specifically, the following SQL statement can be executed: SELECT / *! 40001SQL_NO_CACHE * / order_id FROM T_ORDER FORCE INDEX(`PRIMARY`)ORDERBY order_id DESC LIMIT 1, thereby determining that at least one historical data migration task has been created.

[0105] In this embodiment of the invention, the data migration platform can set a timed scheduling task for at least one historical data migration task, so that data migration can be carried out during off-peak business periods, such as early morning each day.

[0106] In this embodiment of the invention, when the data migration platform determines that the preset timed scheduling task has been started, it obtains the migration status information corresponding to any historical data migration task. Specifically, it can obtain the "PENDING" status from the created historical data migration task, i.e., migrate_task, based on the following SQL statement: SELECT * FROM migrate_task WHERE table_name='T_ORDER'and status='PENDING' order by batch_id asc limit 1.

[0107] In this embodiment of the invention, when the data migration platform determines that the obtained migration status information is in a state to be migrated, it determines the migration start data and migration end data corresponding to any historical data migration task. Furthermore, it can lock the data between the migration start data and the migration end data based on a shared lock to obtain any historical data migration task whose data cannot be modified.

[0108] Furthermore, processing is performed on any historical data migration task that cannot be modified. Once the processing is complete, the shared lock set on any historical data migration task that cannot be modified is released. When it is determined that the status information to be migrated corresponding to at least one historical data migration task is empty, it is determined that the historical data to be migrated has been migrated and a second table including the historical data to be migrated is obtained.

[0109] In this embodiment, multiple migration SQL statements can be constructed based on each data identifier and a preset strategy in any historical data migration task of unmodifiable data. The preset strategy is to wait for the shared lock in the original table to end before performing migration on any historical data migration task of unmodifiable data. An exclusive lock is set between the migration start data and the migration end data of any historical data migration task of unmodifiable data. Then, the migration task of any historical data of unmodifiable data can be executed based on multiple migration SQL statements.

[0110] Specifically, multiple migration SQL statements can be constructed sequentially to execute historical data migration tasks in batches. These migration SQL statements use the preset strategy of `INSERT LOW_PRIORITY IGNORE…LOCK IN SHARE MODE`. This preset strategy waits for all X locks on the original table to finish before starting the transaction and adds S locks to the records between `batch_start_key` and `batch_end_key` to prevent modifications to this data during the migration process. In actual implementation, because the execution time of this statement is very short, it will not cause concurrency issues to the original table.

[0111] For example, the migration SQL statement can be represented as: INSERT LOW_PRIORITY IGNORE INTO T_ORDER_1('order_id','user_id',…) SELECT 'order_id','user_id'… FROM T_ORDERFORCE INDEX(`PRIMARY`) WHERE order_id>=#batch_start_key and order_id<=#batch_end_key and user_id%32=1 LOCK IN SHARE MODE.

[0112] Step 404: Combine the first and second partitions to obtain the corresponding target partition, which includes incremental data and historical data to be migrated.

[0113] In this embodiment of the invention, after obtaining the first and second partition tables, the first and second partition tables can be combined to obtain a corresponding target partition table including incremental data and historical data to be migrated. Further, the target partition table is sent to the corresponding device so that the device can select to use the target partition table or the original table based on selection information.

[0114] like Figure 5 As shown, the present invention provides a data migration apparatus, which includes a receiving unit 501 for receiving migration instruction information; wherein the migration instruction information includes at least an established partitioned table, an original table storing data, and the amount of migration data for each historical data migration; a first processing unit 502 for establishing an incremental data migration task based on the established partitioned table, determining the migration logic corresponding to the incremental data migration task, and executing the incremental data migration task based on the migration logic to obtain a first partitioned table including incremental data; the migration logic is used to determine the incremental data between the original table and the partitioned table, and to synchronize the incremental data; a second processing unit 503 for creating at least one historical data migration task based on the data stored in the original table, the amount of migration data during each historical data migration, and a preset creation rule, executing the at least one historical data migration task to obtain a second partitioned table including historical data to be migrated; and an obtaining unit 504 for combining the first partitioned table and the second partitioned table to obtain a corresponding target partitioned table including incremental data and historical data to be migrated.

[0115] Optionally, the migration logic includes at least: a data coloring sub-logic and an open database modification log sub-logic; the coloring sub-logic is used to determine whether the data stored in the original table has been modified after being migrated to the partitioned table; the open database modification log sub-logic is used to record the addition and deletion information of each piece of data.

[0116] Optionally, the first processing unit 502 is specifically configured to: add two corresponding data coloring fields to both the original table and the partitioned table; based on the two corresponding data coloring fields and the data coloring sub-logic in the migration logic, determine the incremental data that is different between the original table and the partitioned table, and migrate the incremental data to the corresponding partitioned table to obtain an initial first partitioned table; set the original table and the initial first partitioned table to open the MySQL binary log using row format, and listen to the MySQL binary log to extract the data addition and deletion information corresponding to the original table and the initial first partitioned table; the MySQL binary log includes each operation performed on the original table and the initial first partitioned table, as well as the data added, deleted, and updated based on the operation; translate the data addition and deletion information to obtain the data corresponding to the data addition and deletion information, and determine the data corresponding to the data addition and deletion information as the data included in the corresponding first partitioned table to obtain the first partitioned table.

[0117] Optionally, the first processing unit 502 is specifically configured to: determine the first data as an incremental data when it is determined that the first data coloring field and the second data coloring field of the first data in the MySQL binary log corresponding to the partitioned table are consistent and are in a preset format; or, determine the second data as an incremental data when it is determined that the timestamp corresponding to the first data coloring field of the second data in the original table is later than the timestamp corresponding to the second data coloring field of the second data; or, determine the third data as a duplicate incremental data when it is determined that the third data in the original table has not been modified, and the first data coloring field corresponding to the third data is consistent with the second data coloring field corresponding to the third data and is not in the preset format.

[0118] Optionally, the second processing unit 503 is specifically used for: obtaining the smallest data identifier in the data stored in the original table, and assigning the smallest data identifier to the initial migration start variable; using the amount of migration data during each data migration as the loop variable for cyclic migration data, and based on the loop variable, obtaining the last data record of a single cycle migration and the next data record of the last data record from the original table, determined by the amount of migration data as the dividing criterion; when two data records are obtained, determining the migration start variable and migration end variable for the next cycle based on the two data records and the amount of migration data; creating a corresponding historical data migration task based on the migration start variable and the migration end variable; wherein, the migration start variable is used to characterize the data identifier corresponding to the first data record in each sub-table; the migration end variable is used to characterize the data identifier corresponding to the last data record in each sub-table; if no two data records are obtained, it is determined that the historical data migration task has been created, and at least one historical data migration task is obtained.

[0119] Optionally, the second processing unit 503 is specifically configured to: when it is determined that the start time of the preset timed scheduling task has been reached, obtain migration status information corresponding to any of the historical data migration tasks; when it is determined that the obtained migration status information is in a state to be migrated, determine the migration start data and migration end data corresponding to any of the historical data migration tasks, and lock the data between the migration start data and the migration end data based on a shared lock to obtain any of the historical data migration tasks with unmodifiable data; perform processing on any of the historical data migration tasks with unmodifiable data, and when the processing is completed, release the shared lock set on any of the historical data migration tasks with unmodifiable data; when it is determined that the state information to be migrated corresponding to at least one historical data migration task is empty, determine that the historical data to be migrated has been migrated and obtain a second sub-table including the historical data to be migrated.

[0120] Optionally, the second processing unit 503 is specifically configured to: construct multiple migration SQL statements based on each data identifier and a preset strategy in any of the historical data migration tasks of the unmodifiable data; wherein the preset strategy is to wait for the shared lock in the original table to end before performing migration on any of the historical data migration tasks of the unmodifiable data; and to set an exclusive lock between the migration start data and the migration end data of any of the historical data migration tasks of the unmodifiable data; and to execute any of the historical data migration tasks of the unmodifiable data based on the multiple migration SQL statements.

[0121] Optionally, the apparatus further includes a sending unit, configured to: send the target sub-table to a corresponding device, so that the device selects to use the target sub-table or the original table based on selection information.

[0122] This invention provides a computer device, including a program or instructions, which, when executed, are used to perform a data migration method and any optional method provided in this invention.

[0123] This invention provides a storage medium including a program or instructions, which, when executed, are used to perform a data migration method and any optional method provided in this invention.

[0124] Finally, it should be noted that those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0125] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0126] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0127] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A data migration method, characterized in that, The method includes: Receive migration instruction information; wherein, the migration instruction information includes at least the established partitioned table, the original table storing the data, and the amount of migration data for each historical data migration; Based on the established table partitions, an incremental data migration task is created, the migration logic corresponding to the incremental data migration task is determined, and the incremental data migration task is executed based on the migration logic to obtain a first table including incremental data. The migration logic is used to determine the incremental data between the original table and the partition table, and to synchronize the incremental data. The migration logic includes at least: a data coloring sub-logic and an open database modification log sub-logic. The coloring sub-logic is used to determine whether the data stored in the original table has been modified after being migrated to the partition table. The open database modification log sub-logic is used to record the addition and deletion information of each piece of data. Based on the data stored in the original table, the amount of data migrated each time historical data migration, and the preset creation rules, at least one historical data migration task is created, the at least one historical data migration task is executed, and a second sub-table including historical data to be migrated is obtained. The first and second partitions are combined to obtain the corresponding target partition, which includes incremental data and historical data to be migrated. The step of executing the incremental data migration task based on the migration logic to obtain a first partition table including incremental data includes: Add two corresponding data coloring fields to both the original table and the partitioned table; Based on the two corresponding data coloring fields and the data coloring sub-logic in the migration logic, the incremental data that is different between the original table and the partition table is determined, and the incremental data is migrated to the corresponding partition table to obtain the initial first partition table. The original table and the initial first partition table are set to open the MySQL binary log using row format, and the MySQL binary log is monitored to extract the data addition and deletion information corresponding to the original table and the initial first partition table; the MySQL binary log includes each operation performed on the original table and the initial first partition table, as well as the data added, deleted, and updated based on the operation; The data addition and deletion information is translated to obtain the data corresponding to the data addition and deletion information, and the data corresponding to the data addition and deletion information is determined as the data included in the corresponding first sub-table to obtain the first sub-table.

2. The method as described in claim 1, characterized in that, Based on the data coloring field and the data coloring sub-logic in the migration logic, the incremental data that differs between the original table and the partitioned table is determined, including: If it is determined that the first and second data coloring fields of the first data in the MySQL binary log corresponding to the partitioned table are consistent and follow a preset format, then the first data is determined to be an incremental data record; or, If the timestamp corresponding to the first data coloring field of the second data in the original table is later than the timestamp corresponding to the second data coloring field of the second data, then the second data is determined to be an incremental data record synchronized from the partition table to the original table; or, If it is determined that the third data in the original table has not been modified, and the first data coloring field corresponding to the third data is consistent with the second data coloring field corresponding to the third data and is not in the preset format, then the third data is determined as a duplicate incremental data.

3. The method as described in claim 1, characterized in that, Based on the data stored in the original table, the amount of data migrated during each historical data migration, and preset creation rules, a historical data migration task is created, resulting in at least one historical data migration task, including: Obtain the smallest data identifier from the data stored in the original table, and assign the smallest data identifier to the initial migration start variable; The amount of migrated data during each historical data migration is used as the loop variable for the cyclic migration data. Based on the loop variable, the last data of a single cyclic migration and the next data of the last data are obtained from the original table, which are determined by the amount of migrated data as the dividing criterion. When two data records are obtained, the migration start variable and migration end variable for the next cycle are determined based on the two data records and the amount of migration data. Based on the migration start variable and the migration end variable, a corresponding historical data migration task is created; wherein, the migration start variable is used to represent the data identifier corresponding to the first piece of data in each sub-table; the migration end variable is used to represent the data identifier corresponding to the last piece of data in each sub-table; If no two data records are obtained, it is determined that the historical data migration task has been created and at least one historical data migration task has been obtained.

4. The method as described in claim 1, characterized in that, Executing at least one historical data migration task yields a second partition table containing historical data to be migrated, including: When the start time of the preset timed scheduling task is determined, the migration status information corresponding to any of the historical data migration tasks is obtained; When it is determined that the obtained migration status information is in a state to be migrated, the migration start data and migration end data corresponding to any of the historical data migration tasks are determined, and the data between the migration start data and the migration end data is locked based on a shared lock to obtain any of the historical data migration tasks whose data cannot be modified. Process any of the historical data migration tasks for the unmodifiable data, and release the shared lock set on any of the historical data migration tasks for the unmodifiable data after the process is completed; When it is determined that the migration status information corresponding to the at least one historical data migration task is empty, it is determined that the historical data to be migrated has been migrated and a second sub-table including the historical data to be migrated is obtained.

5. The method as described in claim 4, characterized in that, The processing of any of the historical data migration tasks for the unmodifiable data includes: Based on the data identifiers and preset strategies in any of the historical data migration tasks of the immutable data, multiple migration SQL statements are constructed; wherein, the preset strategy is to wait for the shared lock in the original table to end before performing migration on any of the historical data migration tasks of the immutable data; and to set an exclusive lock between the migration start data and the migration end data of any of the historical data migration tasks of the immutable data. Based on the multiple migration SQL statements, execute any of the historical data migration tasks for the unmodifiable data.

6. The method as described in claim 1, characterized in that, After obtaining the target partition table, the method further includes: The target table is sent to the corresponding device so that the device can choose to use the target table or the original table based on the selection information.

7. A data migration device, characterized in that, The device includes: A receiving unit is used to receive migration instruction information; wherein the migration instruction information includes at least the established partitioned table, the original table storing the data, and the amount of migration data for each historical data migration; The first processing unit is configured to establish an incremental data migration task based on the established partitioned table, determine the migration logic corresponding to the incremental data migration task, and execute the incremental data migration task based on the migration logic to obtain a first partitioned table including incremental data. The migration logic is used to determine the incremental data between the original table and the partitioned table, and to synchronize the incremental data. The migration logic includes at least: a data coloring sub-logic and an open database modification log sub-logic. The coloring sub-logic is used to determine whether the data stored in the original table has been modified after being migrated to the partitioned table. The open database modification log sub-logic is used to record the addition and deletion information of each piece of data. The second processing unit is used to create at least one historical data migration task based on the data stored in the original table, the amount of data migrated each time, and the preset creation rules, and to execute the at least one historical data migration task to obtain a second sub-table including historical data to be migrated. The obtaining unit is used to combine the first partition table and the second partition table to obtain the corresponding target partition table including incremental data and historical data to be migrated; The first processing unit is specifically used to add two corresponding data coloring fields to both the original table and the partitioned table; Based on the two corresponding data coloring fields and the data coloring sub-logic in the migration logic, the incremental data that is different between the original table and the partition table is determined, and the incremental data is migrated to the corresponding partition table to obtain the initial first partition table. The original table and the initial first partition table are set to open the MySQL binary log using row format, and the MySQL binary log is monitored to extract the data addition and deletion information corresponding to the original table and the initial first partition table; the MySQL binary log includes each operation performed on the original table and the initial first partition table, as well as the data added, deleted, and updated based on the operation; The data addition and deletion information is translated to obtain the data corresponding to the data addition and deletion information, and the data corresponding to the data addition and deletion information is determined as the data included in the corresponding first sub-table to obtain the first sub-table.

8. A computer device, characterized in that, Includes a program or instructions, which, when executed, cause the method described in any one of claims 1 to 6 to be performed.

9. A storage medium, characterized in that, Includes a program or instructions, which, when executed, cause the method described in any one of claims 1 to 6 to be performed.

Images