Database migration breakpoint resuming method and device, equipment and storage medium

Abstract

The invention discloses a database migration breakpoint resuming method and device, equipment and a storage medium, and belongs to the technical field of databases. According to the database migration breakpoint resuming method, a unique identifier is given to each piece of data according to the transaction of each piece of data, the sequence of the data in the transaction and the sequence of the data in the rowevent, and in the collection process, the collected data and collection sites of the collected data are cached respectively instead of being directly sent to a target end database, so that the data collection efficiency is improved, and the data collection cost is reduced. When a data migration interruption fault occurs, data collection continues from the cached latest collection site, and repeated data filtering is performed by using the data unique identifier, so that the problem of data repeated collection during restarting of the data migration interruption fault is solved, and compared with the prior art, data migration continuing is more accurate and more reliable.

Description

Technical Field

[0001] The present invention relates to the field of database data, and in particular to a database migration breakpoint resume method, device, equipment and storage medium. Background Art

[0002] Database data migration is the process of migrating data from one database to another in full or incremental form. Data migration has very high requirements. The most important thing is to ensure that data is not lost and that the source data and the target data are completely consistent. At the same time, databases usually interact with each other all the time, and data migration cannot affect the business involved in the database. Therefore, data migration often has time requirements. The stability and high availability of data migration are a guarantee for the stable and efficient completion of data migration projects.

[0003] There are too many incidental factors that affect the data migration process, such as server anomalies, breakpoints, network fluctuations, human error operations, etc. These factors often lead to interruptions in the data migration process. There are two scenarios for interruptions: 1. The data migration tool does not stop working after the interruption. In this scenario, the migration tool needs to retry to establish a connection with the source or target end and continue to migrate data; 2. The data migration tool stops working after the interruption. In this scenario, the migration tool needs to be restarted. The migration tool needs to identify the location of the last migration and continue to migrate data. The high availability function of these two scenarios is an indispensable function of the current migration tool, and the latter is currently the most difficult to solve. Summary of the invention

[0004] The purpose of the present invention is to provide a method and device for accurately continuing data collection failure interruption for source-side MYSQL incremental data migration, so as to improve the high availability function of data migration, avoid re-migration or data modification, reduce synchronization time cost and labor cost, and ensure that the business cutover window is not affected.

[0005] To achieve the purpose of the present invention, the present application embodiment first provides a database migration breakpoint resume method, which is characterized by comprising: Collect data from the source database; Cache the data collected from the source database and the collection location corresponding to the data respectively; When a data migration interruption occurs: Get the unique identifier of the latest cached data and the latest collection location of the cache; Use the latest collection location as the collection starting point to continue collecting data from the source database; The unique identifier of the data to be collected is compared with the unique identifier of the latest data to determine whether the data is duplicate data; When the data that continues to be collected is duplicate data, it is filtered out until non-duplicate data appears, and the data and the collection location corresponding to the data are cached respectively; Continue to collect data from the source database, and cache the collected data and the collection locations corresponding to the data, until the data collection of the source database is completed; Send all cached data collected from the source database to the target database.

[0006] To achieve the purpose of the present invention, the embodiment of the present application further provides a database migration device, which is characterized by comprising: The acquisition module is used to collect data from the source database; Data relay module, used to cache data collected from the source database; A location cache module is used to cache the collection locations corresponding to the data collected from the source database; The interruption processing module is used to handle the following when a data migration interruption occurs: Obtain the unique identifier of the latest data cached in the data relay module and the latest collection location cached in the location cache module; The latest acquired collection location is sent to the collection module, instructing the collection module to continue collecting data from the source database using the latest collection location as the collection starting point; Compare the unique identifier of the data that continues to be collected by the collection module with the unique identifier of the latest data to determine whether the data is duplicate data; When the data that continues to be collected is duplicate data, it is filtered out until non-duplicate data appears, and the data relay module and the location cache module are instructed to cache the data and the collection location corresponding to the data respectively; Instruct the acquisition module to continue to collect data from the source database, and cache the collected data and the collection sites corresponding to the data through the data relay module and the site cache module respectively until the data collection of the source database is completed; The playback module is used to send all the data collected from the source database and cached in the data relay module to the target database after the data collection of the source database is completed.

[0007] To achieve the purpose of the present invention, an embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the memory stores a program that can be run on the processor, and is characterized in that when the program is executed by the processor, the steps of the above-mentioned database migration breakpoint resumption method are implemented.

[0008] Also, a computer-readable storage medium is provided, storing at least one program, characterized in that the at least one program can be executed by at least one processor to implement the steps of the above-mentioned database migration breakpoint resumption method.

[0009] Compared with the prior art, the present invention has the following beneficial effects: 1. Improved the high availability of resynchronization caused by failures during data migration.

[0010] 2. Avoid re-migration, greatly reducing time and labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of a database migration breakpoint resume method flow in an embodiment of the present application; Figure 2 A schematic diagram of the unique identification composition of data in an embodiment of the present application; Figure 3 This is a schematic diagram of the flow chart of the duplicate data comparison and filtering method according to an embodiment of the present application; Figure 4 A schematic diagram of the structure of a database migration device according to an embodiment of the present application; Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0012] In order to make the technical solution of the present invention more clearly understood, the technical solution of the present invention is further described in detail below according to specific embodiments and in combination with the accompanying drawings. Obviously, the following embodiments are only part of the embodiments of the present invention and do not constitute a limitation on the technical solution of the present invention. For those of ordinary skill in the art, other embodiments can also be obtained according to the embodiments of the present application without paying creative labor.

[0013] It should be noted that the order of steps described in the method embodiments in the following embodiments does not constitute a limitation on the technical solution of the present invention. For ordinary technicians in this field, without paying any creative labor, they can also reasonably adjust the order of certain steps according to the embodiments of the present application. Example 1

[0014] This embodiment provides a database migration breakpoint resumption method, which is applied to the incremental data migration scenario where the source database is MYSQL, and aims to improve the high availability of data migration and ensure that data migration can be continued with low cost loss even if an interruption failure occurs during the migration process.

[0015] like Figure 1 As shown, the database migration breakpoint resume method of this embodiment includes the following steps: S101. Collect data from a source database.

[0016] S102: Cache the data collected from the source database and the collection locations corresponding to the data respectively.

[0017] In this embodiment, the data collected from the source database is not sent directly to the target database, but is first cached in a specific data relay module (named kafka module in this embodiment) in the order of collection. At the same time, the collection sites corresponding to the data are cached in sequence in a specific storage module (named GTID_SET module in this embodiment). In this embodiment, the MYSQL database global transaction ID, i.e. GTID, is used as the data collection site, and the value of the collection site increases with the transaction ID.

[0018] S103. When a data migration interruption failure occurs, the unique identifier of the latest cached data is obtained from the kafka module, and the latest collection location corresponding to the latest data is obtained from the GTID_SET module.

[0019] In this embodiment, each incremental data of MYSQL is uniquely identified as follows: In MYSQL, GTID, as the MYSQL data global transaction ID, can give each transaction a unique identifier, and gradually increases as the transaction occurs. Therefore, this embodiment uses the GTID of the transaction to which the data belongs as the first identifier of the data according to the transaction order in which each data occurs. A transaction may have multiple data, and the data in the transaction is ordered. Each transaction has multiple row_events, so this embodiment gives the data a second identifier EVENT_ID according to the order of the row_event to which the data belongs. Each row_event also has multiple data, and the data is given a third identifier ROW_ID according to the order in which the data occurs in the row_event.

[0020] like Figure 2 As shown, in this embodiment, the unique identifier of the data includes a first identifier, a second identifier and a third identifier.

[0021] The first identifier indicates the order in which the transactions to which the data belongs occur. It is directly represented by GTID and is the unique identifier of the global transaction of the data itself. It increases incrementally and is used to compare the order of transactions. It is the highest priority identifier and its value is determined by the database.

[0022] The second identifier indicates the order in which the row_event to which the data belongs occurs in the transaction to which it belongs, and is represented by EVENT_ID. Its value is numbered according to the order of row_event in the transaction, starting from 1.

[0023] The third identifier indicates the order in which the data occurs in the row_event to which it belongs, and is represented by ROW_ID. Its value is numbered according to the order in the data row_event, starting from 1.

[0024] The uniqueness of the data is guaranteed through the calibration of the above unique identifier.

[0025] S104: continue to collect data from the source database by taking the latest acquired collection location as the collection starting point.

[0026] S105, filtering duplicate data, including: Compare the unique identifier of the data to be collected with the unique identifier of the latest data obtained from the Kafka module to determine whether the data is duplicate data; When the data that continues to be collected is duplicate data, it is filtered out until non-duplicate data appears, indicating that the data has not been collected. The data is directly cached in the Kafka module, and the collection location corresponding to the data is cached in the GTID_SET module.

[0027] like Figure 3 As shown, the method for determining whether data is repeated by comparing the unique identifier of the data is as follows: First, compare the first identifier GTID of the data. If it is smaller than the GTID of the latest data, it means that it has been collected and is duplicate data. If it is equal to the GTID of the latest data, continue to compare the second identifier EVENT_ID of the data. Compare EVENT_ID. If it is smaller than the EVENT_ID of the latest data, it means that it has been collected and is duplicate data. If it is equal to the EVENT_ID of the latest data, continue to compare the third identifier ROW_ID of the data. Compare ROW_ID. If it is less than or equal to the ROW_ID of the latest data, it means that it has been collected and is duplicate data.

[0028] S106. Continue to collect data from the source database, cache the data to the Kafka module, and cache the collection location corresponding to the data to the GTID_SET module until the source database data collection is completed.

[0029] S107. When the source database data collection is completed, all the data collected from the source database and cached in the Kafka module are sent to the target database.

[0030] The database migration breakpoint resumption method of this embodiment gives each data a unique identifier according to the transaction in which each data occurs, the order of the data in the transaction, and the order of the data in row_event. During the collection process, the collected data and its collection location are cached separately instead of sending the collected data directly to the target database. When a data migration interruption failure occurs, data collection continues from the latest collection location in the cache, and duplicate data is filtered using the data unique identifier, thereby solving the problem of duplicate data collection when restarting a data migration interruption failure. Compared with the prior art, data migration continuation is more accurate and reliable. Example 2

[0031] This embodiment provides a database migration device, such as Figure 4 As shown, including: The collection module is used to collect data from the source database.

[0032] The data relay module Kafka is used to cache data collected from the source database.

[0033] The site cache module GTID_SET is used to cache the collection sites corresponding to the data collected from the source database.

[0034] The interruption processing module is used to handle the following when a data migration interruption occurs: Get the unique identifier of the latest data cached in the data relay module kafka and the latest collection location cached in the location cache module GTID_SET; Send the latest collection location to the collection module, instructing the collection module to continue collecting data from the source database using the latest collection location as the collection starting point; Compare the unique identifier of the data that the collection module continues to collect with the unique identifier of the latest data obtained to determine whether the data is duplicate data; When the data that continues to be collected is duplicate data, it is filtered out until non-duplicate data appears, and the data relay module kafka and the site cache module GTID_SET are instructed to cache the data and the collection site corresponding to the data respectively; Instruct the collection module to continue collecting data from the source database, and cache the collected data and the collection site corresponding to the data through the data relay module kafka and the site cache module GTID_SET respectively until the data collection of the source database is completed.

[0035] The playback module is used to pull all the data collected from the source database from the data relay module kafka and write it into the target database after the data collection of the source database is completed.

[0036] In this embodiment, the data collected by the collection module from the source database is not sent directly to the target database, but first cached to the data relay module kafka, and the collection site corresponding to the collected data is cached to the site cache module GTID_SET. The GTID_SET module is a high-availability module, which is used to cache the collection sites corresponding to the collected data during the data collection phase and to provide a collection starting point when restarting due to a failure. In this embodiment, the MYSQL database global transaction ID, i.e., GTID, is used as the data collection site, and the value of the collection site increases with the transaction ID.

[0037] When an exception occurs in incremental data migration and the collection is restarted for repair, the unique identifier of the latest data is first obtained from Kafka, and then data is continued to be collected according to the latest collection location cached by GTID_SET. The collected data is not sent directly to Kafka, but the unique identifier of the data is compared with the unique identifier of the latest data in Kafka, and the data unique identifier is used to filter out duplicate data. Only non-duplicate data is sent to Kafka for caching, which solves the problem of duplicate data collection when data migration is interrupted and restarted, and ensures accurate data continuation.

[0038] In this embodiment, the unique identifier of the data includes a first identifier, a second identifier and a third identifier.

[0039] The first identifier indicates the order in which the transactions to which the data belongs occur. It is directly represented by GTID and is the unique identifier of the global transaction of the data itself. It increases incrementally and is used to compare the order of transactions. It is the highest priority identifier and its value is determined by the database.

[0040] The second identifier indicates the order in which the row_event to which the data belongs occurs in the transaction to which it belongs, and is represented by EVENT_ID. Its value is numbered according to the order of row_event in the transaction, starting from 1.

[0041] The third identifier indicates the order in which the data occurs in the row_event to which it belongs, and is represented by ROW_ID. Its value is numbered according to the order in the data row_event, starting from 1.

[0042] The uniqueness of the data is guaranteed through the calibration of the above unique identifier. Example 3

[0043] This embodiment provides an electronic device, such as Figure 5 As shown, it includes a processor 31 and a memory 32 in communication with the processor 31. The memory 32 stores a program that can be run on the processor 31, and when the program is executed by the processor 31, the steps of the database migration breakpoint resume method provided in the above embodiment are implemented.

[0044] The above program can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. Example 4

[0045] This embodiment provides a computer-readable storage medium, which stores at least one program. The at least one program can be executed by at least one processor to implement the steps of the database migration breakpoint resume method provided in the above embodiment.

[0046] The above storage media include: U disk, mobile hard disk, read-only memory, random access memory, disk or optical disk and other media that can store program code and their combination. With the development of science and technology, the meaning of storage media may become more and more extensive and not limited to tangible media.

[0047] Those skilled in the art can understand that the above-mentioned embodiments are specific embodiments for implementing the present invention. In addition to the above-mentioned embodiments, the present invention may also have other implementation modes; any technical solutions formed by equivalent replacement or equivalent transformation shall fall within the protection scope required by the present invention.

Claims

1. A database migration breakpoint resume method, characterized in that: include: Collect data from the source database; Cache the data collected from the source database and the collection location corresponding to the data respectively; When a data migration interruption occurs: Get the unique identifier of the latest cached data and the latest collection location of the cache; Use the latest collection location as the collection starting point to continue collecting data from the source database; The unique identifier of the data to be collected is compared with the unique identifier of the latest data to determine whether the data is duplicate data; When the data that continues to be collected is duplicate data, it is filtered out until non-duplicate data appears, and the data and the collection location corresponding to the data are cached respectively; Continue to collect data from the source database, and cache the collected data and the collection locations corresponding to the data, until the data collection of the source database is completed; Send all cached data collected from the source database to the target database.

2. The database migration breakpoint resume method according to claim 1, characterized in that: The source database is MYSQL.

3. The database migration breakpoint resume method according to claim 1, characterized in that: The unique identifier includes a first identifier, a second identifier and a third identifier. The first identifier indicates the order in which the transaction to which the data belongs occurs, the second identifier indicates the order in which the row_event to which the data belongs occurs in the transaction to which it belongs, and the third identifier indicates the order in which the data occurs in the row_event to which it belongs.

4. The database migration breakpoint resume method according to claim 3, characterized in that: The unique identifier of the data to be continuously collected is compared with the unique identifier of the latest data to determine whether the data is duplicate data, including: when the value of the first identifier of the data is less than the value of the first identifier of the latest data, determining that the data is duplicate data.

5. The database migration breakpoint resume method according to claim 3, characterized in that: The unique identifier of the data to be continuously collected is compared with the unique identifier of the latest data to determine whether the data is duplicate data, including: when the value of the first identifier of the data is equal to the value of the first identifier of the latest data and the value of the second identifier of the data is less than the value of the second identifier of the latest data, the data is determined to be duplicate data.

6. The database migration breakpoint resume method according to claim 3, characterized in that: The unique identifier of the data to be continuously collected is compared with the unique identifier of the latest data to determine whether the data is duplicate data, including: when the values ​​of the first identifier and the second identifier of the data are respectively equal to the values ​​of the first identifier and the second identifier of the latest data and the value of the third identifier of the data is less than the value of the third identifier of the latest data, the data is determined to be duplicate data.

7. The database migration breakpoint resume method according to claim 1, characterized in that: The collection location is determined according to the occurrence order of the transaction to which the data belongs.

8. A database migration device, characterized in that: include: The acquisition module is used to collect data from the source database; Data relay module, used to cache data collected from the source database; A location cache module is used to cache the collection locations corresponding to the data collected from the source database; The interruption processing module is used to handle the following when a data migration interruption occurs: Obtain the unique identifier of the latest data cached in the data relay module and the latest collection location cached in the location cache module; The latest acquired collection location is sent to the collection module, instructing the collection module to continue collecting data from the source database using the latest collection location as the collection starting point; Compare the unique identifier of the data that continues to be collected by the collection module with the unique identifier of the latest data to determine whether the data is duplicate data; When the data that continues to be collected is duplicate data, it is filtered out until non-duplicate data appears, and the data relay module and the location cache module are instructed to cache the data and the collection location corresponding to the data respectively; Instruct the acquisition module to continue to collect data from the source database, and cache the collected data and the collection sites corresponding to the data through the data relay module and the site cache module respectively until the data collection of the source database is completed; The playback module is used to send all the data collected from the source database and cached in the data relay module to the target database after the data collection of the source database is completed.

9. An electronic device comprising a processor and a memory, wherein the memory stores a program that can be run on the processor, characterized in that: When the program is executed by the processor, the steps of the database migration breakpoint resumption method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing at least one program, characterized in that: The at least one program can be executed by at least one processor to implement the steps of the database migration breakpoint resumption method described in any one of claims 1-7.

Images