Database migration method and related device

Abstract

The embodiment of the invention provides a database migration method and a related device.The database migration method comprises the steps that in the database migration process, total migration is conducted firstly, and in the total migration process, data migration is conducted on at least two pieces of target data corresponding to a table with a foreign key cascade relation according to the same snapshot site, therefore, the data of the target database and the data of the source database after full migration are consistent. And then incremental migration is carried out, and the initial site of incremental playback is determined according to the snapshot sites corresponding to the at least two target data in the incremental migration process, so that the accuracy and efficiency of incremental playback are improved. At least two pieces of target data are placed in one thread queue through incremental playback, so that at least two targets in a target database can successfully trigger cascade operation, and the consistency of the target database and the data of the target database is achieved.

Description

Technical Field

[0001] This application relates to the field of database technologies, and in particular, to a database migration method and related devices. Background Art

[0002] With the development of computer technologies, databases have been widely used for data storage. During the process of adjusting and optimizing services related to the stored data, it is often necessary to migrate databases. Database migration is to transfer data from one database to another, for example, to transfer data from a database management system (such as My Structured Query Language, MySQL) 1 to a database management system 2. Database migration generally includes two migration methods: full migration and incremental migration. Full migration is to migrate the data of the source database to the target database at one time. After the full migration is completed, the data newly added or modified in the source database since the full migration is synchronized to the target database through incremental migration.

[0003] If the tables to which the data in the database migration belong have foreign key cascade relationships, foreign key cascade means that the foreign key in one table is associated with the primary key in another table. For example, when a record in the parent table is deleted or updated, the corresponding records in the related child table will also be deleted or updated. However, during the incremental migration process, due to the change in the order of multiple migrated data or the setting of system cascade parameters, when a record in the parent table is deleted or updated, the corresponding records in the related child table cannot be deleted or updated accordingly, resulting in inconsistent data between the target database and the source database. Summary of the Invention

[0004] This application provides a database migration method and related devices. By performing single-threaded incremental playback on at least two data corresponding to tables with foreign key cascade relationships, at least two data in the target database trigger the foreign key cascade operations in the target database in order, so that the data in the source database and the target database are consistent.

[0005] To achieve the above object, this application adopts the following technical solutions:

[0006] In a first aspect, a database migration method is provided. The method includes:

[0007] Obtain the data to be migrated in the source database and determine at least two target data among the data to be migrated. At least two of the target data belong to tables with foreign key cascade relationships; migrate at least two of the target data from the source database to the target database in full amount according to the same snapshot position; after the full migration is completed, determine the starting position of incremental playback according to the snapshot position, and perform single-threaded incremental playback on at least two of the target data according to the starting position.

[0008] Thus, during the database migration process, a full-scale migration is first performed. During the full-scale migration, at least two target data corresponding to tables with foreign key cascade relationships are migrated according to the same snapshot position, so that the data in the target database and the source database are consistent after the full-scale migration. Then, an incremental migration is performed. During the incremental migration, the starting position of the incremental playback is determined based on the snapshot positions corresponding to at least two target data, so as to improve the accuracy and efficiency of the incremental playback. By placing at least two target data in a thread queue during the incremental playback, at least two targets in the target database can be successfully triggered for cascade operations, thereby achieving the consistency of the data between the target database and the target database.

[0009] In some embodiments of the first aspect, the single-threaded incremental playback of at least two target data according to the starting position includes: determining the trigger order of the cascade operations corresponding to at least two target data according to the log of the source database, and performing single-threaded incremental playback of at least two target data according to the trigger order and the starting position.

[0010] Thus, the trigger order of the cascade operations of the target data is determined through the log of the source data, and the target data in the target database is triggered according to the trigger order corresponding to the cascade operations of the source database, so as to achieve the consistency between the source database and the target database.

[0011] In some embodiments of the first aspect, the single-threaded incremental playback of at least two target data according to the starting position includes: obtaining the foreign key cascade parameters of each cascade operation corresponding to at least two target data, where the foreign key cascade parameters are used to indicate whether to enable foreign key constraint checking; performing single-threaded incremental playback of at least two target data according to the foreign key cascade parameters of each cascade operation and the starting position.

[0012] Thus, by setting the corresponding foreign key cascade parameters for each cascade operation, when performing cascade operations on at least two target data, the control of foreign key constraints can be carried out with the cascade operation as the basic control unit, improving the granularity and precision of the control of foreign key constraints during the database migration process.

[0013] In some embodiments of the first aspect, the data to be migrated includes at least two first target data and at least two second target data. The tables to which at least two of the first target data belong have a first foreign key cascade relationship, and the tables to which at least two of the second target data belong have a second foreign key cascade relationship. The single-threaded incremental playback of at least two of the target data according to the starting site includes: performing single-threaded incremental playback of at least two of the first target data according to the starting site; performing single-threaded incremental playback of at least two of the second target data according to the starting site.

[0014] In this way, since the tables to which multiple data in the data to be migrated may belong to multiple different foreign key cascade relationships, at least two target data corresponding to each foreign key cascade relationship can be incrementally played back in a single thread. The multiple target data corresponding to the tables with different foreign key cascade relationships are respectively placed in different threads for incremental playback to achieve multi-threaded parallel incremental playback and improve the efficiency of incremental playback.

[0015] In some embodiments of the first aspect, the data to be migrated includes at least two first target data and at least two second target data. The tables to which at least two of the first target data belong have a first foreign key cascade relationship, and the tables to which at least two of the second target data belong have a second foreign key cascade relationship. The full migration of at least two of the target data from the source database to the target database according to the same snapshot site includes: fully migrating at least two of the first target data from the source database to the target database at a first snapshot site; fully migrating at least two of the second target data from the source database to the target database at a second snapshot site.

[0016] If the amount of data of the target data in the data to be migrated is large, then fully migrating all the target data according to the same snapshot site requires storing a large amount of data at the same time, occupying a large amount of storage space and having a high requirement for the storage performance of the device. Since at least two of the target data in the data to be migrated may belong to multiple different foreign key cascade relationships, the target data corresponding to the tables with the same foreign key cascade relationship can be fully migrated according to the same snapshot site. The target data corresponding to the tables with different foreign key cascade relationships are fully migrated according to different snapshot sites. Thus, the requirement for the storage performance of the device can be reduced and the consistency of the target data corresponding to the tables with different foreign key cascade relationships can be achieved.

[0017] In some embodiments of the first aspect, the full - volume migration of at least two pieces of the target data from the source database to the target database according to the same snapshot site includes: locking the tables where at least two pieces of the target data are located; obtaining a repeatable - read snapshot of at least two pieces of the target data; and migrating the snapshot data at the current snapshot site from the source database to the target database in full volume.

[0018] In some embodiments of the first aspect, the full - volume migration of at least two pieces of the target data from the source database to the target database according to the same snapshot site includes: obtaining the order of at least two pieces of the target data; and migrating at least two pieces of the target data from the source database to the target database in full volume according to the same snapshot site based on the order.

[0019] In a second aspect, a database migration device is provided, including: a memory including computer - readable instructions; and a processor communicating with the memory, where the processor is configured to execute the computer - readable instructions so that the database migration device executes the database migration method according to any one of the first aspect.

[0020] In a third aspect, a computer - readable storage medium is provided, including a program or instructions that, when executed by a processor, implement the database migration method according to any one of the first aspect.

[0021] In a fourth aspect, a chip is provided, including a processor configured to call and run instructions stored in a memory from the memory so that a database migration device installed with the chip executes the database migration method according to any one of the first aspect.

[0022] For the beneficial effects brought by each possible implementation manner in the database migration method provided in the second aspect, the database migration device provided in the third aspect, the computer - readable storage medium provided in the fourth aspect, and the chip provided in the fifth aspect of the embodiments of the present application, reference may be made to the descriptions in various possible implementation manners of the first aspect, and details are not described herein one by one. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 FIG. is a schematic diagram of a database migration scenario provided by an embodiment of the present application;

[0024] Figure 2 FIG. is a flowchart of a database migration method provided by an embodiment of the present application;

[0025] Figure 3 FIG. is a schematic diagram of a full - volume migration scenario provided by an embodiment of the present application;

[0026] Figure 4 FIG. is a schematic diagram of a full - volume migration scenario provided by an embodiment of the present application;

[0027] Figure 5 A schematic diagram of a full - volume migration scenario provided by an embodiment of the present application;

[0028] Figure 6 A schematic diagram of a full - volume migration scenario provided by an embodiment of the present application;

[0029] Figure 7 A schematic diagram of an incremental migration scenario provided by an embodiment of the present application;

[0030] Figure 8 A schematic diagram of an incremental migration scenario provided by an embodiment of the present application;

[0031] Figure 9 A schematic diagram of the structure of a database migration device provided by an embodiment of the present application. Detailed implementation manners

[0032] Next, the technical solutions in the present application will be described in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all of the embodiments.

[0033] First, introduce the technical terms involved in the embodiments of the present application:

[0034] 1. Foreign key referential

[0035] The foreign key referential relationship means that in a database, the association relationship between tables is defined through foreign key constraints. The cascade operations include cascade update and cascade delete. For example, if a foreign key constraint defines a parent table and a child table, when a record in the parent table is updated or deleted, the relevant records in the child table are automatically updated or deleted. Cascade update means that when the primary key of the parent table is updated, the corresponding foreign key in the child table will also be updated. Cascade delete means that when a record in the parent table is deleted, the corresponding record in the child table will also be automatically deleted.

[0036] The database is mainly used for data storage, management, maintenance, etc. When the business related to the stored data changes or is adjusted, it is often necessary to migrate the database. Database migration is to migrate data from one database to another, for example, from the enterprise's MySQL database to the cloud MySQL database. Database migration usually includes two stages: the full - volume stage and the enhancement stage. Among them, in the full - volume stage, the entire database is migrated from the source database to the target database. After the full - volume migration, incremental migration starts. Incremental migration is to migrate the data that has changed after the full - volume migration. If the tables to which the data to be migrated belong have foreign - key cascade relationships, when the data in one of the tables with foreign - key cascade relationships changes, the data in the other table will also change accordingly. In the incremental migration stage, due to reasons such as the inconsistent order of the data in the target database and the data in the source database or system - set parameters, when updating or deleting data in the parent table, the update or deletion operation of the data in the child table will not be triggered, resulting in inconsistent data between the source database and the target database.

[0037] Based on the above problems, the embodiments of this application provide a database migration method. First, migrate the data to be migrated from the source database to the target database through full - volume migration. During the full - volume migration process, at least two target data in the data to be migrated are fully migrated according to the same snapshot site, where at least two target data belong to tables with foreign - key cascade relationships; after the full - volume migration is completed, determine the starting site of incremental playback according to the snapshot sites of at least two target data, and perform single - thread incremental playback on at least two target data according to the starting site, so as to enable at least two target data to trigger cascade operations in the target database and make the data in the source database and the target database consistent.

[0038] Please refer to Figure 1 , Figure 1 which is a schematic diagram of a database migration scenario provided by the embodiments of this application. When the business changes, the user can extract data from the source database through Structured Query Language (SQL) and import the extracted data into the target database.

[0039] Database migration includes full - volume migration and incremental migration. During the full - volume migration process, at least two target data corresponding to the tables with foreign - key cascade relationships in the data to be migrated are fully migrated according to the same snapshot site; for example, Figure 1 if table1 and table2 in have foreign - key cascade relationships, then when fully migrating data1 of table1 and data2 of table2 to the target database, both data1 of table1 and data2 of table2 are fully migrated according to snapshot site 1.

[0040] After the full-scale migration is completed, determine the starting site of incremental playback based on the snapshot sites corresponding to at least two target data. Perform single-threaded incremental playback on at least two target data based on the starting site. As Figure 1 shown, place the data 1 of table1 and the data 2 of table2 in the queue of thread 1. That is, perform single-threaded incremental playback on at least two target data corresponding to tables with the same foreign key cascade relationship, so that at least two target data can successfully perform cascade operations, realizing the data consistency between the source database and the target database.

[0041] It is easy to understand that Figure 1 the source database and the target database in may be relational databases, or other databases that can store tables with foreign key cascade relationships.

[0042] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of a database migration method provided by an embodiment of the present application. Figure 2 The database migration method in includes: S201~S203.

[0043] S201. Obtain the data to be migrated in the source database, and determine at least two target data in the data to be migrated. The tables to which the at least two target data belong have foreign key cascade relationships.

[0044] It is easy to understand that database migration is to migrate the data in one database to another database. The data to be migrated can be all the data of the source database or part of the data in the source database.

[0045] Among them, the foreign key is used to constrain the relationship between different data. A table is a structure for organizing data, which consists of rows and columns. The rows in the table contain specific data records, such as the data to be migrated. Data is the specific information stored in the table, which can be various types such as numbers, texts, dates, times, etc. The parent table and the child table refer to two tables with a foreign key relationship. Among them, the table with the foreign key is called the child table, and the table containing the primary key referenced by the foreign key is called the parent table. The foreign key is a field in the child table that references the primary key field in the parent table. The tables with foreign key cascade relationships can be the same table, that is, the parent table and the child table are the same table. For example, the data in column 1 and column 2 of table1 are the foreign key and the primary key respectively; the tables with foreign key cascade relationships can also be two different tables, that is, the parent table and the child table are different tables respectively. For example, the primary key in table1 and the foreign key in table2. Of course, the tables with foreign key cascade relationships can also include at least three different tables.

[0046] In some embodiments, if at least two target data belong to different columns of the same table, that is, the tables to which at least two target data belong have a foreign key cascade relationship; if at least two target data belong to columns of at least two different tables, then the at least two tables to which at least two target data belong have a foreign key cascade relationship.

[0047] S202. Migrate all the at least two target data from the source database to the target database at the same snapshot point.

[0048] Optionally, when migrating all the data from the source database to the target database, first obtain the snapshot data of the data in the source database at a specific snapshot point; then perform a full migration from the source database to the target database through the snapshot data. Among them, the snapshot point is the state of the database at a certain specific time point. When performing database migration, the snapshot point can be used to record the state of the data at a certain time point and perform a full migration based on this state.

[0049] Optionally, since the tables to which at least two target data belong have a foreign key cascade relationship, perform a full migration of the at least two target data corresponding to the tables with a foreign key cascade relationship at the same snapshot point to keep the at least two target data consistent with each other.

[0050] Optionally, if the source database is MySQL, the snapshot point can be a binary log (binlog) point. The binlog point refers to the position marker in the binlog file, which is used to identify the position in the binlog file.

[0051] Optionally, if the at least two tables to which at least two target data belong include at least two tables, multiple table snapshots can be used to create snapshot data of multiple tables in the source database. Here, the multiple tables are the tables to which at least two target data belong.

[0052] S203. After the full migration is completed, determine the starting point of incremental playback based on the snapshot points corresponding to the at least two target data, and perform single-threaded incremental playback on the at least two target data based on the starting point.

[0053] Optionally, after the full migration is completed, incremental migration is required. Incremental migration is to synchronize the newly added or modified data after the full migration to the target database. Determine the starting point of incremental playback based on the snapshot points of the at least two target data, so that the newly added or modified data after the full migration can be synchronized to the target database through incremental playback, improving the efficiency and accuracy of data migration, and avoiding the situation that setting the starting point too early affects the efficiency of incremental migration or setting the starting point too late affects the accuracy of data migration. Of course, in other embodiments, the starting point can also be adjusted according to actual needs. For example, the starting point can be set before or after the snapshot point according to actual needs.

[0054] Optionally, since the tables to which at least two target data belong have a foreign key cascade relationship, at least two target data corresponding to the tables with the foreign key cascade relationship are placed in the same thread queue for incremental playback; so that during the incremental playback process, the cascade operations of at least two target data in the target database can be successfully triggered, thereby achieving data consistency between the source database and the target database.

[0055] Optionally, at least two target data can be incrementally replayed in a single thread according to the incremental log and the starting site. Since the incremental log is a record of newly added or modified data during the data migration process; after the full-scale migration is completed, the newly added or modified data is synchronized to the target database through the incremental log. By reading the data in the incremental log, the data can be quickly applied to the target database without scanning the entire data set again. This can greatly reduce the incremental migration time and resource consumption and improve the efficiency of data incremental migration.

[0056] In this way, during the database migration process, first perform a full-scale migration. During the full-scale migration process, at least two target data corresponding to the tables with a foreign key cascade relationship are migrated according to the same snapshot site, so that the data in the target database and the source database are consistent after the full-scale migration. Then perform an incremental migration. The starting site of the incremental playback is determined according to the snapshot positions corresponding to at least two target data during the incremental migration process to improve the accuracy and efficiency of the incremental playback. By placing at least two target data in a thread queue during the incremental playback, at least two targets in the target database can successfully trigger the cascade operation, thereby achieving data consistency between the target database and the target database.

[0057] It is easy to understand that the tables to which at least two target data in the data to be migrated belong may belong to multiple different foreign key cascade relationships. For example, as Figure 3 shown, Figure 3 A schematic diagram of a full-scale migration scenario provided by an embodiment of the present application. Figure 3 In it, data 1 belongs to table 1, data 2 belongs to table 2, and table 1 and table 2 have a foreign key cascade relationship 1; data 3 belongs to table 3, data 4 belongs to table 4, and table 3 and table 4 have a foreign key cascade relationship 2. Then data 1, data 2, data 3, and data 4 can be fully migrated according to the same snapshot site; for example, obtain the snapshot data of data 1, data 2, data 3, and data 4 at snapshot site 1; then perform a full-scale migration from the source database to the target database based on the data at snapshot site 1.

[0058] Optionally, since multiple data may belong to multiple different tables, a multi-table snapshot method can be used to obtain the snapshot data of multiple data at the same time.

[0059] It is easy to understand that if the amount of target data in the data to be migrated is large, then migrating all the target data in full according to the same snapshot site requires storing a large amount of data at the same time, occupying a large amount of storage space, and having high requirements for the storage performance of the device. Since at least two tables to which the target data belongs in the data to be migrated may belong to multiple different foreign key cascade relationships, the target data corresponding to the tables with the same foreign key cascade relationship can be migrated in full according to the same snapshot site. The target data corresponding to the tables with different foreign key cascade relationships is migrated in full according to different snapshot sites. For example, as Figure 4 shown Figure 4 is a schematic diagram of a full migration scenario provided by an embodiment of the present application; Figure 4 in, Data 1 belongs to Table 1, Data 2 belongs to Table 2, and Table 1 and Table 2 have a foreign key cascade relationship 1; Data 3 belongs to Table 3, Data 4 belongs to Table 4, and Table 3 and Table 4 have a foreign key cascade relationship 2; then the snapshot data of Data 1 and Data 2 at snapshot site 1 can be obtained; the snapshot data of Data 3 and Data 4 at snapshot site 2 can be obtained; then, according to the snapshot data at snapshot site 1, Data 1 and Data 2 are migrated in full from the source database to the target database, and according to the snapshot data at snapshot site 2, Data 3 and Data 4 are migrated in full from the source database to the target database. Thus, the requirements for the storage performance of the device can be reduced and the consistency of the target data corresponding to the tables with different foreign key cascade relationships can be achieved.

[0060] Optionally, during the full migration process, in order to make the order of at least two target data in the source database the same as the order of at least two target data in the target database, the order of at least two target data in the source database can be obtained, and according to this order, at least two target data are migrated in full from the source database to the target database according to the same snapshot site. It is easy to understand that if at least two target data are serially transmitted in a single thread according to the determined order, the order of at least two target data received by the target database can be the same as the order of at least two target data in the source database. However, since the amount of data migrated in full is large, in order to improve the efficiency of the full migration, multi-threaded concurrent transmission can be used for the data that needs to be migrated in full. In this way, at least two target data can be serially transmitted in a single thread according to the order, but the data to be migrated is transmitted in parallel by multiple threads, that is, the order of at least two target data in the source database can be the same as the order of at least two target data in the target database, and the data transmission efficiency can be improved.

[0061] It is easy to understand that if at least two tables to which the target data belongs in the data to be migrated may belong to multiple different foreign key cascade relationships, for example, Figure 5 is a schematic diagram of a full migration scenario provided by an embodiment of the present application; Figure 5Data 1 belongs to Table 1, data 2 belongs to Table 2, and Table 1 and Table 2 have a foreign key cascade relationship 1; data 3 belongs to Table 3, data 4 belongs to Table 4, and Table 3 and Table 4 have a foreign key cascade relationship 2. Then, during the full-scale migration process, data 1, data 2, data 3, and data 4 can be fully migrated through Thread 1. Of course, in other embodiments, if at least two target data among the data to be migrated may belong to multiple different foreign key cascade relationships, then at least two target data corresponding to tables with the same foreign key cascade relationship can be fully migrated in a single thread, and at least two target data corresponding to tables with different foreign key cascade relationships can be fully migrated in multiple threads. For example Figure 6 is a schematic diagram of a full-scale migration scenario provided by an embodiment of the present application; Figure 6 In it, data 1 belongs to Table 1, data 2 belongs to Table 2, and Table 1 and Table 2 have a foreign key cascade relationship 1; data 3 belongs to Table 3, data 4 belongs to Table 4, and Table 3 and Table 4 have a foreign key cascade relationship 2. Then, during the full-scale migration process, data 1 and data 2 are fully migrated through Thread 1; data 3 and data 4 are fully migrated in Thread 2.

[0062] In some embodiments, during the full-scale migration process, the tables to which at least two target data belong can be locked; these tables cannot be written during the locking period to prevent modification operations on the source database during the migration process, resulting in data inconsistency; then, a repeatable read snapshot of at least two target data is obtained; the snapshot data at the current snapshot position is fully migrated from the source database to the target database. The data after locking the tables is snapshotted across multiple tables to obtain a repeatable snapshot, and the full-scale migration is performed through the repeatable snapshot.

[0063] It is easy to understand that if at least two target data among the data to be migrated may belong to multiple different foreign key cascade relationships, for example, Figure 7 is a schematic diagram of an incremental migration scenario provided by an embodiment of the present application; Figure 7 In it, data 1 belongs to Table 1, data 2 belongs to Table 2, and Table 1 and Table 2 have a foreign key cascade relationship 1; data 3 belongs to Table 3, data 4 belongs to Table 4, and Table 3 and Table 4 have a foreign key cascade relationship 2. Then, during the incremental migration process, data 1, data 2, data 3, and data 4 can be incrementally replayed through Thread 1. That is, during the enhanced migration process, the target data corresponding to the tables with foreign key cascade relationships is incrementally replayed in a single thread, that is, all target data is placed in a thread queue for incremental replay.

[0064] Of course, in other embodiments, if the data to be migrated includes at least two first target data and at least two second target data, the tables to which the at least two first target data belong have a first foreign key cascade relationship, and the tables to which the at least two second target data belong have a second foreign key cascade relationship; the single-threaded incremental playback of the at least two target data according to the starting site in S203 includes: incrementally playing back the at least two first target data through a first thread according to the starting site corresponding to the first target data; incrementally playing back the at least two second target data through a second thread according to the starting site corresponding to the second target data. For example, Figure 8 is a schematic diagram of a scenario of incremental migration provided by an embodiment of the present application; Figure 8 In the figure, data 1 belongs to Table 1, data 2 belongs to Table 2, and Table 1 and Table 2 have a foreign key cascade relationship 1; data 3 belongs to Table 3, data 4 belongs to Table 4, and Table 3 and Table 4 have a foreign key cascade relationship 2. Then, during the incremental migration process, data 1 and data 2 can be incrementally played back through Thread 1, and data 3 and data 4 can be incrementally played back through Thread 2. Since the tables to which multiple data in the data to be migrated may belong to multiple different foreign key cascade relationships, at least two target data corresponding to each foreign key cascade relationship can be incrementally played back in a single thread, and multiple target data corresponding to tables with different foreign key cascade relationships are respectively placed in different threads for incremental playback to achieve multi-threaded parallel incremental playback and improve the efficiency of incremental playback.

[0065] It is easy to understand that there may be multiple cascade operations after the full-scale migration, such as update, delete, and delete. That is, after the full-scale migration, at least two target data in the source database sequentially execute three cascade operations: update, delete, and delete. The type of the cascade operation and the order of the multiple cascade operations are recorded in the log of the source database. If the log can be a binary log (Binary Log), a transaction log (Transaction Log), or a general log file (General Log), then during the incremental playback process, the multiple cascade operations and the order of the multiple cascade operations of the target data corresponding to the tables with the same foreign key cascade relationship can be determined through the log of the source database, and then the multiple cascade operations are executed according to the log of the source database. Thus, at least two target data in the target database are triggered in the same cascade operation order as the source database to achieve the consistency between the source database and the target database.

[0066] Optionally, for each cascading operation of at least two target data corresponding to tables with the same foreign key cascading relationship, the triggering order is related to the type of cascading operation. For example, cascading deletion means that when the data in the parent table is deleted, the data in the related child tables will also be deleted; cascading update means that when the records in the parent table are updated, the records in the related child tables will also be updated. To achieve the consistency between the source database and the target database, the triggering order of each cascading operation of the target data in the target database should be the same as that in the source database, and the triggering order of each cascading operation of the target data can be obtained from the log of the source database. Then, the single-threaded incremental playback of at least two target data based on the starting site in S203 includes: determining the triggering order of the cascading operations corresponding to at least two target data according to the log of the source database, and performing single-threaded incremental playback of at least two target data according to the triggering order and the starting site. Determine the triggering order of the cascading operations of the target data through the log of the source data, and trigger the target data in the target database according to the triggering order corresponding to the cascading operation of the source database to achieve the consistency between the source database and the target database.

[0067] It is easy to understand that a database transaction refers to a series of database operations executed as a single logical unit of work, and each transaction may include one or more cascading operations. If the target database is MySQL, the foreign key cascade parameter (foreign_key_checks) is a system variable used to control the checking of foreign key constraints during data operation execution. When foreign_key_checks is set to 1, it means that foreign key constraint checking is enabled; when set to 0, it means that foreign key constraint checking is disabled. That is, during the incremental playback process of the target data corresponding to tables with the same foreign key cascade relationship, multiple cascading operations need to be executed, such as deleting and updating operations in sequence. Due to user requirements or actual business requirements, each cascading operation can correspond to different foreign key cascade parameters. For example, when performing a delete operation, foreign_key_checks is set to 1, that is, foreign key constraint checking is enabled; when performing an update operation, foreign_key_checks is set to 0, that is, foreign key constraint checking is not enabled. If the target database is of other types, it is also possible to indicate whether to enable foreign key constraint checking through corresponding foreign key cascade parameters. Then, in S203, single-threaded incremental playback of at least two target data is performed according to the starting point, including: obtaining the foreign key cascade parameters of each cascading operation corresponding to at least two target data, where the foreign key cascade parameters are used to indicate whether to enable foreign key constraint checking; performing single-threaded incremental playback of at least two target data according to the foreign key cascade parameters of each cascading operation and the starting point. In this way, by setting corresponding foreign key cascade parameters for each cascading operation, when performing cascading operations on at least two target data, the control of foreign key constraints can be carried out with the cascading operation as the basic control unit, improving the granularity and accuracy of foreign key constraint control during the database migration process.

[0068] Optionally, the foreign key cascade parameters of the cascading operations of the transaction corresponding to at least two target data can be determined by parsing the log of the source database, so that during incremental playback, cascading operations can be performed on at least two target data of the target database according to the foreign key cascade parameters corresponding to each cascading operation. That is, before executing the Data Manipulation Language (DML), by obtaining the foreign key cascade parameters of the cascading operations of the transaction corresponding to at least two target data, refined control of the target data can be achieved.

[0069] It is easy to understand that during the full - volume stage and the incremental stage of database migration, the target data in the data to be migrated can be determined so as to perform a full - volume migration of the target data according to the same snapshot position. In order to determine at least two target data in the data to be migrated, first, the table where one of the target data is located is obtained as the starting table, and then the metadata of the starting table is obtained. The database table names of the foreign - key child table and the foreign - key parent table corresponding to the table are determined through the metadata. Then, based on the starting table as the starting point, with the foreign - key child table and the foreign - key parent table in the metadata as the foreign - key cascade conditions, a depth - first traversal is performed on the data to be migrated to determine other tables that have a cascade relationship with the starting table. Then, the starting table and other tables (which may be one table or multiple tables) have a foreign - key cascade relationship, and the foreign keys and primary keys in the tables with a foreign - key cascade relationship are the target data in the data to be migrated.

[0070] It is easy to understand that in addition to target data, the data to be migrated may also include non - target data. The tables to which the non - target data belongs do not have a foreign - key cascade relationship. Then, during the full - volume migration and incremental migration of the non - target data, the transmission method of the non - target data is not limited, and the transmission method can be adjusted according to actual needs.

[0071] It can be understood that in the above - mentioned embodiments, a database migration method provided by an embodiment of the present application, during the database migration process, first performs a full - volume migration. During the full - volume migration, at least two target data corresponding to the tables with a foreign - key cascade relationship are migrated according to the same snapshot position, so that the data in the target database and the source database is consistent after the full - volume migration. The tables to which at least two target data in the data to be migrated belong may belong to multiple different foreign - key cascade relationships, and the target data corresponding to the tables with different foreign - key cascade relationships can be migrated according to different snapshot positions during the full - volume migration;

[0072] Then, an incremental migration is performed. During the incremental migration, the starting position of the incremental playback is determined based on the snapshot positions corresponding to at least two target data to improve the accuracy and efficiency of the incremental playback. The tables to which at least two target data in the data to be migrated belong may belong to multiple different foreign - key cascade relationships; multiple target data corresponding to the tables with different foreign - key cascade relationships are respectively placed in different threads for incremental playback to achieve multi - thread parallel incremental playback to improve the efficiency of the incremental playback; and during the incremental playback, at least two target data are placed in a thread queue so that at least two targets in the target database can successfully trigger cascade operations, thereby achieving the consistency of the data in the target database and the target database.

[0073] It should be understood that the above is only to help those skilled in the art better understand the embodiments of the present application, rather than to limit the scope of the embodiments of the present application. Those skilled in the art can obviously make various equivalent modifications or changes according to the above examples. For example, some steps in the above methods are not necessary, or some steps can be newly added, etc. Or any combination of any two or any more of the above embodiments. The solutions after such modifications, changes or combinations also fall within the scope of the embodiments of the present application.

[0074] It should also be understood that the ways, situations, categories and the division of embodiments in the embodiments of the present application are only for the convenience of description and should not constitute special limitations. The features in various ways, categories, situations and embodiments can be combined without contradiction.

[0075] It should also be understood that the various digital numbers involved in the embodiments of the present application are only for the convenience of distinction in description and do not limit the scope of the embodiments of the present application. The size of the serial numbers of the above processes does not mean the sequence of execution. The execution sequence of each process should be determined by its function and internal logic and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0076] It should also be understood that the above description of the embodiments of the present application focuses on emphasizing the differences between the various embodiments. The same or similar parts not mentioned can be referred to each other. For the sake of brevity, they will not be elaborated here.

[0077] The above combination Figures 1-8 has described the embodiments of the method and system provided by the embodiments of the present application. Next, the database migration device provided by the embodiments of the present application will be described.

[0078] In this embodiment, the functional modules of the database migration device can be divided according to the above method. For example, corresponding to each function, it can be divided into each functional module, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is illustrative and is only a logical function division. There may be other division methods in actual implementation.

[0079] It should be noted that the relevant content of each step involved in the above method embodiment can be cited to the functional description of the corresponding functional module and will not be elaborated here.

[0080] The database migration device provided by the embodiments of the present application is used to execute the database migration method provided by the above method embodiment, so the same effect as the above implementation method can be achieved.

[0081] In other embodiments, in the case of adopting an integrated unit, the database migration device may include a processing module, a storage module, and a communication module. Among them, the processing module may be used to control and manage the operations of the database migration device. For example, it may be used to support the database migration device in executing the steps performed by the processing unit. The storage module may be used to support the storage of program codes, data, etc. The communication module may be used to support the communication between the database migration device and other network devices and the database migration device itself.

[0082] Among them, the processing module may be a processor or a controller. It may be used to implement or execute various exemplary logic blocks, modules, and circuits described in connection with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processing (DSP) and a microprocessor, and so on. The storage module may be a memory. Specifically, the communication module may be a device that interacts with other database migration devices or network devices, such as a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, etc.

[0083] Based on the same concept, an embodiment of this application also provides a database migration device. Refer to Figure 9 , Figure 9 which shows a schematic structural diagram of an exemplary database migration device of this application. Figure 9 The shown database migration device can execute the steps in any database migration method performed by the database migration device provided in the embodiments of this application.

[0084] This database migration device 900 includes at least one processor 901, a memory 903, and at least one network interface 904.

[0085] The processor 901 is, for example, a general-purpose CPU, a digital signal processor (DSP), a network processor (NP), a GPU, a neural network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits or application-specific integrated circuits (ASICs) for implementing the solution of this application, a programmable logic device (PLD), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The PLD is, for example, a complex programmable logic device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can implement or execute various logic blocks, modules, and circuits described in connection with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and so on.

[0086] Optionally, the database migration device 900 further includes a bus 902. The bus 902 is used to transfer information between the components of the database migration device 900. The bus 902 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 902 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0087] The memory 903 is, for example, a read only memory (ROM) or other type of storage device that can store static information and instructions, or a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read only memory (EEPROM), a compact disc read only memory (CD ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 903 exists independently, for example, and is connected to the processor 901 through the bus 902. The memory 903 can also be integrated with the processor 901.

[0088] The network interface 904 uses any device such as a transceiver to communicate with other devices or communication networks, and the communication network can be an Ethernet, a radio access network (RAN) or a wireless local area network (WLAN), etc. The network interface 904 can include a wired network interface and can also include a wireless network interface. Specifically, the network interface 904 can be an Ethernet interface, such as a fast Ethernet (FE) interface, a gigabit Ethernet (GE) interface, an asynchronous transfer mode (ATM) interface, a WLAN interface, a cellular network interface or a combination thereof. The Ethernet interface can be an optical interface, an electrical interface or a combination thereof. In some embodiments of the present application, the network interface 904 can be used for the database migration device 900 to communicate with other devices.

[0089] In a specific implementation, as some embodiments, the processor 901 can include one or more CPUs. Each of these processors can be a single-core processor or a multi-core processor. Here, the processor can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0090] In a specific implementation, as some embodiments, the database migration apparatus 900 may include multiple processors. Each of these processors may be a single-core processor or a multi-core processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0091] In some embodiments, the memory 903 is used to store program instructions for executing the solution of this application, and the processor 901 may execute the program instructions stored in the memory 903. That is, the database migration apparatus 900 may implement the method provided by the method embodiment shown in the above embodiment through the program instructions in the processor 901 and the memory 903. The program instructions may include one or more software modules. Optionally, the processor 901 itself may also store the program instructions for executing the solution of this application.

[0092] In the specific implementation process, the processor 901 in the database migration apparatus 900 of this application reads the instructions in the memory 903, so that Figure 9 the database migration apparatus 900 shown can execute all or part of the steps in the database migration method executed by the database migration apparatus in the above embodiment.

[0093] Among them, each step of the method described in the above embodiment is completed by the integrated logic circuit of the hardware in the processor of the database migration apparatus 900 or the instructions in the form of software. The steps of the method embodiment disclosed in combination with this application can be directly embodied as being executed and completed by the hardware processor, or executed and completed by a combination of the hardware and software modules in the processor. The software module may be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method embodiment. To avoid repetition, it will not be described in detail here.

[0094] It should be understood that the above-mentioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. It is worth noting that the processor can be a processor supporting the advanced RISC machines (ARM) architecture.

[0095] Furthermore, in an alternative embodiment, the above-mentioned memory can include a read-only memory and a random access memory, and provide instructions and data to the processor. The memory can also include a non-volatile random access memory. For example, the memory can also store information about the device type.

[0096] The memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM).

[0097] The database migration device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effects are similar, which will not be elaborated here.

[0098] This application embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in the above method embodiment is implemented.

[0099] This application embodiment also provides a computer program product. When the computer program product runs on the database migration device, the database migration device is enabled to execute the method described in the above method embodiment when executed.

[0100] This application embodiment provides a chip, including a processor, which is used to call and run instructions stored in a memory, so that a communication device installed with the chip executes the method described in any database migration device provided in this application embodiment.

[0101] This application embodiment also provides a chip system, including a processor, where the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the method described in the above method embodiment. Wherein, the chip system can be a single chip or a chip module composed of multiple chips.

[0102] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in this application embodiment are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, or magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0103] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by relevant hardware instructed by a computer program. This program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The aforementioned storage medium can include various media that can store program codes, such as ROM, random access memory (RAM), magnetic disks, or optical discs.

[0104] In this application, the naming or numbering of steps does not mean that the steps in the method process must be executed in the time / logical sequence indicated by the naming or numbering. The already named or numbered process steps can be changed in the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0105] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0106] In the embodiments provided in this application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the devices or units can be in an electrical, mechanical or other form.

[0107] It should be understood that in the description of this application specification and the appended claims, the terms "include", "comprise", "have" and any variations thereof are intended to cover non-exclusive inclusion, all meaning "including but not limited to", unless otherwise specifically emphasized in other ways. For example, a process, method, system, product or equipment that includes a series of steps or modules does not have to be limited to those steps or modules clearly listed, but can include other steps or modules not clearly listed or inherent to these processes, methods, products or equipment.

[0108] In the description of this application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship. For example, A / B can represent A or B. The "and / or" in this application is used to describe the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Here, A and B can be singular or plural.

[0109] Moreover, in the description of the present application, unless otherwise specified, "a plurality of" means two or more than two. "At least one of the following" or its similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c may represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, c may be single or multiple.

[0110] As used in the specification of the present application and the appended claims, the term "if" may be interpreted as "when", "once", "in response to determining", or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]" depending on the context.

[0111] In addition, in the description of the specification of the present application and the appended claims, terms such as "first", "second", etc. are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments described here can be implemented in an order other than that shown or described here; features defined with "first" and "second" may explicitly or implicitly include at least one such feature.

[0112] In the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design solution described as "exemplarily" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplarily" or "for example" is intended to present relevant concepts in a specific manner.

[0113] Reference to "one embodiment" or "some embodiments" etc. described in the specification of the present application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A database migration method, characterized in that, The method includes: Obtain the data to be migrated in the source database and determine at least two target data among the data to be migrated, where the tables to which at least two of the target data belong have a foreign key cascade relationship; Migrate at least two of the target data in full volume from the source database to the target database according to the same snapshot position; After the full-volume migration is completed, determine the starting position of the incremental playback based on the snapshot position, and perform single-threaded incremental playback on at least two of the target data based on the starting position.

2. The method according to claim 1, wherein The performing single-threaded incremental playback on at least two of the target data based on the starting position includes: Determine the trigger order of the cascade operations corresponding to at least two of the target data according to the log of the source database, and perform single-threaded incremental playback on at least two target data based on the trigger order and the starting position.

3. The method according to claim 1 or 2, characterized in that, The performing single-threaded incremental playback on at least two of the target data based on the starting position includes: Obtain the foreign key cascade parameters of each cascade operation corresponding to at least two of the target data, where the foreign key cascade parameters are used to indicate whether to enable foreign key constraint checking; Perform single-threaded incremental playback on at least two of the target data based on the foreign key cascade parameters of each cascade operation and the starting position.

4. The method according to any one of claims 1 to 3, characterized in that The data to be migrated includes at least two first target data and at least two second target data, the tables to which at least two of the first target data belong have a first foreign key cascade relationship, and the tables to which at least two of the second target data belong have a second foreign key cascade relationship; The performing single-threaded incremental playback on at least two of the target data based on the starting position includes: Perform single-threaded incremental playback on at least two of the first target data based on the starting position; Perform single-threaded incremental playback on at least two of the second target data based on the starting position.

5. The method according to any one of claims 1 to 4, characterized in that The data to be migrated includes at least two first target data and at least two second target data, the tables to which at least two of the first target data belong have a first foreign key cascade relationship, and the tables to which at least two of the second target data belong have a second foreign key cascade relationship; The migrating at least two of the target data in full volume from the source database to the target database according to the same snapshot position includes: Migrate at least two of the first target data in full volume from the source database to the target database at the first snapshot position; Migrate at least two of the second target data in full volume from the source database to the target database at the second snapshot position.

6. The method according to any one of claims 1 to 5, characterized in that, The migrating at least two of the target data in full volume from the source database to the target database according to the same snapshot position includes: Lock the tables where at least two of the target data are located; Obtain the repeatable read snapshot of at least two of the target data; Migrate the snapshot data at the current snapshot position in full volume from the source database to the target database.

7. The method according to any one of claims 1 to 6, characterized in that The migrating at least two of the target data in full volume from the source database to the target database according to the same snapshot position includes: Obtain the order of at least two of the target data; Migrate at least two of the target data in full volume from the source database to the target database according to the said order at the same snapshot site.

8. A database migration device, characterized in that, It includes: A memory, which includes computer-readable instructions; A processor communicatively connected to the memory, and the processor is configured to execute the computer-readable instructions, so that the database migration device executes the database migration method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, It includes a program or instructions, which when executed by a processor, implement the database migration method according to any one of claims 1-7.

10. A chip, characterized in that, It includes a processor, which is configured to call and run the instructions stored in the memory from the memory, so that the database migration device installed with the chip executes the database migration method according to any one of claims 1-7.

Images