Data migration method and device, equipment and medium

Abstract

The invention provides a data migration method. The method can be applied to the technical fields of big data and financial science and technology. The method comprises the steps of terminating data backup of a second database to a first database, obtaining all data in the second database, and generating first to-be-migrated data. And synchronizing the first to-be-migrated data into a target database to generate first synchronous data. And starting data backup of the second database to the first database, obtaining incremental data in the second database, and generating second to-be-migrated data. And synchronizing the second to-be-migrated data into a target database to generate second synchronous data. And merging the first synchronous data and the second synchronous data to generate target synchronous data, and storing the target synchronous data to a target server to complete data migration. The invention further provides a data migration device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of financial technology, specifically to the field of big data technology, and in particular to a data migration method, device, equipment, medium and program product. Background Art

[0002] With the rapid development of computer technology, enterprises including banks are in urgent need of migrating heterogeneous databases from traditional databases to new databases based on business development and security needs.

[0003] Traditional data migration technology has technical issues such as prolonged business interruption and poor business continuity caused by downtime for migration; data consistency risks; long data verification time affecting business continuity; poor solution universality; and high risk of switchback. Summary of the Invention

[0004] In view of the above problems, the present disclosure provides a data migration method, apparatus, device, medium and program product.

[0005] According to a first aspect of the present disclosure, a data migration method is provided, the method comprising: terminating a data backup of a first database by a second database, obtaining all data in the second database, and generating first data to be migrated; synchronizing the first data to be migrated to a target database, and generating first synchronization data; performing a consistency check on the first data to be migrated and the first synchronization data; if the check is consistent, starting a data backup of the first database by the second database, obtaining incremental data in the second database, and generating second data to be migrated, wherein the incremental data in the second database is the incremental data in the first database backed up by the second database after terminating the data backup of the first database by the second database; synchronizing the second data to be migrated to a target database, and generating second synchronization data; performing a consistency check on the second data to be migrated and the second synchronization data; and if the check is consistent, merging the first synchronization data and the second synchronization data to generate target synchronization data, and storing the target synchronization data to a target server to complete the data migration.

[0006] According to an embodiment of the present disclosure, a consistency check is performed on the first data to be migrated and the first synchronized data, including: obtaining multiple first data tables of the first data to be migrated and multiple second data tables of the first synchronized data; detecting whether the table names of the multiple first data tables and the multiple second data tables are all the same; if the table names of the multiple first data tables and the multiple second data tables are all the same, performing primary key range segmentation on the data of the first data table and the data of the second data table with the same table name, generating n interval data of the data of the first data table and n interval data of the data of the second data table, wherein n is an integer and n is greater than 1; and performing a one-to-one consistency check on the n interval data of the data of the first data table and the n interval data of the data of the second data table.

[0007] According to an embodiment of the present disclosure, a one-to-one consistency check is performed on n interval data of the data of the first data table and n interval data of the data of the second data table, including: using multi-threading to calculate the hash values ​​of the one-to-one corresponding data in the n interval data of the data of the first data table and the n interval data of the data of the second data table; and performing a consistency check through the hash values ​​of the one-to-one corresponding data in the n interval data of the data of the first data table and the n interval data of the data of the second data table.

[0008] According to an embodiment of the present disclosure, a consistency check is performed on the second data to be migrated and the second synchronization data, including: obtaining the data log of the second data to be migrated, performing data conversion, and generating first lightweight exchange format data; obtaining the data log of the second synchronization data, performing data conversion, and generating second lightweight exchange format data; and performing a consistency check on the table name, primary key, data type and field value of the first lightweight exchange format data and the second lightweight exchange format data.

[0009] According to an embodiment of the present disclosure, the first synchronization data and the second synchronization data are merged to generate target synchronization data, including: merging the first synchronization data and the second synchronization data to generate third synchronization data; obtaining all the data of the first database, and performing a consistency check on all the data of the first database and the third synchronization data; if the check is consistent, updating the third synchronization data based on the incremental data of the first database to generate fourth synchronization data, and merging the first synchronization data and the fourth synchronization data to generate target synchronization data.

[0010] According to an embodiment of the present disclosure, a consistency check is performed on all the data of the first database and the third synchronization data, including: obtaining the incremental data log in all the data of the first database and the log of the second synchronization data in the third synchronization data; deleting the heartbeat log and timestamp of the incremental data log in all the data of the first database to generate a first log; deleting the heartbeat log and timestamp of the log of the second synchronization data in the third synchronization data to generate a second log; and performing a consistency check on the first log and the second log.

[0011] According to a second aspect of the present disclosure, a data migration device is provided, which includes: a first generation module, which is used to stop the second database from backing up data of the first database, obtain all data in the second database, and generate first data to be migrated; a second generation module, which is used to synchronize the first data to be migrated to a target database to generate first synchronization data; a first detection module, which is used to perform consistency detection on the first data to be migrated and the first synchronization data; a third generation module, which is used to start the second database to back up data of the first database if the detection is consistent, obtain incremental data in the second database, and generate second data to be migrated, wherein the incremental data in the second database is the incremental data in the first database backed up by the second database after stopping the second database from backing up data of the first database; a fourth generation module, which is used to synchronize the second data to be migrated to the target database to generate second synchronization data; a second detection module, which is used to perform consistency detection on the second data to be migrated and the second synchronization data; and a fifth generation module, which is used to merge the first synchronization data and the second synchronization data if the detection is consistent, to generate target synchronization data, and store the target synchronization data to a target server to complete data migration.

[0012] According to an embodiment of the present disclosure, the first detection module includes: a first acquisition module for acquiring multiple first data tables of the first data to be migrated and multiple second data tables of the first synchronized data; a third detection module for detecting whether the table names of the multiple first data tables and the multiple second data tables are all the same; a sixth generation module for performing primary key range splitting on the data of the first data table and the data of the second data table with the same table name if the table names of the multiple first data tables and the multiple second data tables are all the same, and generating n interval data of the data of the first data table and n interval data of the data of the second data table, wherein n is an integer and n is greater than 1; and a fourth detection module for performing one-to-one consistency detection on the n interval data of the data of the first data table and the n interval data of the data of the second data table.

[0013] According to an embodiment of the present disclosure, the fourth detection module includes: a first calculation module, which is used to use multi-threading to calculate the hash value of the one-to-one corresponding data in the n interval data of the data in the first data table and the n interval data of the data in the second data table; and a fifth detection module, which is used to perform consistency detection through the hash value of the one-to-one corresponding data in the n interval data of the data in the first data table and the n interval data of the data in the second data table.

[0014] According to an embodiment of the present disclosure, the second detection module includes: a seventh generation module, used to obtain the data log of the second data to be migrated, perform data conversion, and generate first lightweight exchange format data; an eighth generation module, used to obtain the data log of the second synchronization data, perform data conversion, and generate second lightweight exchange format data; and a sixth detection module, used to perform consistency detection on the table name, primary key, data type and field value of the first lightweight exchange format data and the second lightweight exchange format data.

[0015] According to an embodiment of the present disclosure, the fifth generation module includes: a ninth generation module, which is used to merge the first synchronization data and the second synchronization data to generate third synchronization data; a seventh detection module, which is used to obtain all the data of the first database and perform consistency detection on all the data of the first database and the third synchronization data; a tenth generation module, which is used to update the third synchronization data based on the incremental data of the first database to generate fourth synchronization data if the detection is consistent; and an eleventh generation module, which is used to merge the first synchronization data and the fourth synchronization data to generate target synchronization data.

[0016] According to an embodiment of the present disclosure, the seventh detection module includes: a second acquisition module, used to obtain the incremental data log in all the data of the first database and the log of the second synchronization data in the third synchronization data; a twelfth generation module, used to delete the heartbeat log and timestamp of the incremental data log in all the data of the first database, and generate a first log; a thirteenth generation module, used to delete the heartbeat log and timestamp of the log of the second synchronization data in the third synchronization data, and generate a second log; and an eighth detection module, used to perform consistency detection on the first log and the second log.

[0017] According to a third aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned data migration method.

[0018] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which executable instructions or computer programs are stored. When the instructions or computer programs are executed by a processor, the processor executes the above-mentioned data migration method.

[0019] According to a fifth aspect of the present disclosure, a computer program product is also provided, including a computer program, which implements the above-mentioned data migration method when executed by a processor.

[0020] The technical solution disclosed in the present invention improves the reliability of migrated data through the technical means of two-stage verification of full data and incremental data and master-slave separation and synchronization, and maximizes business continuity without shutting down the main database; at the same time, it improves the strong consistency before and after data migration, improves the reliability of data migration; and achieves the technical effect of improving detection efficiency through the dual detection mechanism of full data and incremental data. In addition, since the target synchronization data is directly generated to complete the data migration, multiple types of databases can be applied, which enhances universality and reduces the risk of back-cutting. It solves the technical problems of poor business continuity in existing technical solutions; the risk of data consistency; the long data verification time affecting business continuity, poor solution universality and high risk of back-cutting. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0022] Figure 1 The following schematically illustrates an application scenario of the data migration method and apparatus according to an embodiment of the present disclosure;

[0023] Figure 2 The following schematically shows a flow chart of a data migration method according to an embodiment of the present disclosure;

[0024] Figure 3 The flowchart schematically shows a consistency check between the first data to be migrated and the first synchronization data in the data migration method according to an embodiment of the present disclosure;

[0025] Figure 4 A flowchart schematically illustrates a method for performing one-to-one consistency check on n intervals of data in a first data table and n intervals of data in a second data table in a data migration method according to an embodiment of the present disclosure;

[0026] Figure 5 A flowchart schematically illustrates consistency detection between the second data to be migrated and the second synchronization data in the data migration method according to an embodiment of the present disclosure;

[0027] Figure 6A flowchart schematically illustrates merging first synchronization data and second synchronization data to generate target synchronization data in a data migration method according to an embodiment of the present disclosure;

[0028] Figure 7 A flowchart schematically illustrates a method for performing consistency check on all data in a first database and third synchronization data in a data migration method according to an embodiment of the present disclosure;

[0029] Figure 8 The following schematically shows a flow chart of a specific embodiment of the data migration method according to an embodiment of the present disclosure;

[0030] Figure 9 A structural block diagram of a data migration device according to an embodiment of the present disclosure is schematically shown; and

[0031] Figure 10 A block diagram of an electronic device suitable for implementing a data migration method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0032] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0033] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0034] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0035] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0036] The accompanying drawings illustrate some block diagrams and / or flow charts. It should be understood that some blocks in the block diagrams and / or flow charts, or combinations thereof, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable control device, so that when executed by the processor, these instructions may create a device for implementing the functions / operations described in the block diagrams and / or flow charts.

[0037] First, let’s explain the technical terms that appear in this article as follows:

[0038] Heartbeat data: This is the core information of health monitoring mechanisms widely used in distributed systems, network communications, and software and hardware monitoring. It is used to send cautionary signals or messages regularly to indicate that data is normal.

[0039] Heartbeat data: logs used to record heartbeat data behavior.

[0040] An embodiment of the present disclosure provides a data migration method, which includes: terminating the data backup of the first database by the second database, obtaining all the data in the second database, and generating first data to be migrated. Synchronizing the first data to be migrated to the target database to generate first synchronization data. Performing a consistency check on the first data to be migrated and the first synchronization data. If the detection is consistent, starting the data backup of the first database by the second database, obtaining the incremental data in the second database, and generating second data to be migrated, wherein the incremental data in the second database is the incremental data in the first database backed up by the second database after terminating the data backup of the first database by the second database. Synchronizing the second data to be migrated to the target database to generate second synchronization data. Performing a consistency check on the second data to be migrated and the second synchronization data. And if the detection is consistent, merging the first synchronization data and the second synchronization data to generate target synchronization data, and storing the target synchronization data to the target server to complete the data migration.

[0041] According to the embodiments of the present disclosure, through the technical means of two-stage verification of full data and incremental data and master-slave separation and synchronization, the reliability of migrated data is improved, and the main database does not need to be shut down, thereby maximizing business continuity; at the same time, the strong consistency before and after data migration is improved, and the reliability of data migration is improved; and through the dual detection mechanism of full data and incremental data, the technical effect of improving detection efficiency is achieved. Moreover, since the target synchronization data is directly generated to complete the data migration, multiple types of databases can be applied, the universality is enhanced, and the technical effect of reducing the risk of back-cutting is achieved. It solves the technical problems of poor business continuity in existing technical solutions; the existence of data consistency risks; the long data verification time affecting business continuity, the poor universality of the solution, and the high risk of back-cutting.

[0042] Figure 1 The following diagram schematically illustrates an application scenario of the data migration method and device according to an embodiment of the present disclosure. Figure 1 The examples shown are merely examples of scenarios in which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0043] like Figure 1 As shown, the application scenario 100 according to this embodiment may include an application scenario of data migration. A network 104 is used as a medium for providing a communication link between a first terminal device 101, a second terminal device 102, a third terminal device 103, and a server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0044] A user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be configured with a primary server group with a backup server or a target server to be migrated.

[0045] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0046] It should be noted that the data migration method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data migration device provided in the embodiment of the present disclosure can generally be set in the server 105. The data migration method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data migration device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0047] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0048] The following will be based on Figure 1 The scene described by Figures 2 to 8 The data migration method of the disclosed embodiment is described in detail. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0049] Figure 2 The flowchart of the data migration method according to the embodiment of the present disclosure is schematically shown.

[0050] like Figure 2 As shown, the method 200 includes steps S201 to S207.

[0051] Step S201 : terminating the data backup of the first database by the second database, obtaining all the data in the second database, and generating first data to be migrated.

[0052] For example, the second database is a backup database of the source database, and the first database is a primary database of the source database. The first and second databases are primary and backup databases and serve as the databases to be migrated. After the second database terminates its data backup of the first database, all data in the second database is obtained as static data to generate the first data to be migrated.

[0053] Step S202: Synchronize the first data to be migrated to the target database to generate first synchronization data.

[0054] For example, the target database is a target database for data migration, and the static first data to be migrated can be synchronized to the target database through a data interface.

[0055] Step S203: performing consistency check on the first data to be migrated and the first synchronization data.

[0056] Figure 3 The flowchart of performing consistency detection on the first data to be migrated and the first synchronization data in the data migration method according to an embodiment of the present disclosure is schematically shown.

[0057] like Figure 3 As shown, the method 300 includes steps S301 to S304.

[0058] Step S301: Acquire multiple first data tables of the first data to be migrated and multiple second data tables of the first synchronized data.

[0059] Step S302 : Detect whether the table names of the plurality of first data tables and the plurality of second data tables are all the same.

[0060] Step S303: If the table names of the multiple first data tables and the multiple second data tables are all the same, the data of the first data tables and the data of the second data tables with the same table names are split by primary key range to generate n interval data of the data of the first data table and n interval data of the data of the second data table, where n is an integer and n is greater than 1.

[0061] If the multiple first data tables and the multiple second data tables have different table names, the first data tables and the second data tables with different table names are saved, an exception log is generated, and a warning message is generated and sent to the user.

[0062] Step S304 : performing a one-to-one consistency check on the n interval data of the data in the first data table and the n interval data of the data in the second data table.

[0063] Figure 4 The flowchart schematically shows a one-to-one consistency check between n intervals of data in a first data table and n intervals of data in a second data table in a data migration method according to an embodiment of the present disclosure.

[0064] like Figure 4 As shown, the method 400 includes steps S401-S402.

[0065] Step S401 : using multiple threads to calculate hash values ​​of one-to-one corresponding data in n intervals of data in the first data table and n intervals of data in the second data table.

[0066] Step S402 : performing consistency check based on hash values ​​of one-to-one corresponding data in n intervals of data in the first data table and n intervals of data in the second data table.

[0067] If the hash values ​​of the one-to-one corresponding data in the n interval data of the data in the first data table and the n interval data of the data in the second data table are the same, proceed to the next step. If different hash values ​​appear, save them, generate an exception log, and generate a warning message to send to the user.

[0068] Multi-threaded hash value consistency calculations improve detection efficiency and accuracy, while also saving computer and network resources. Table name detection and primary key range segmentation improve consistency detection efficiency and reliability.

[0069] Return to reference Figure 2 In step S204, if the detection is consistent, the second database is started to back up the data of the first database, the incremental data in the second database is obtained, and the second data to be migrated is generated, wherein the incremental data in the second database is the incremental data in the first database backed up by the second database after the second database terminates the data backup of the first database.

[0070] For example, the incremental data in the second database can be a segment of incremental data in the first database that was backed up by the second database during the period from when the second database terminated its backup of the first database to when the second database started backing up the first database. The incremental data in the second database can also be multiple segments of incremental data in the first database that were backed up by the second database during the period from when the second database terminated its backup of the first database to when the second database started backing up the first database, within preset time segments. Each segment of incremental data has the same preset time segment, and in this case, the incremental data increases with the preset time segment. In this case, the incremental data in the second database is time-dynamic data.

[0071] Step S205: Synchronize the second data to be migrated to the target database to generate second synchronized data.

[0072] Step S206: performing consistency check on the second data to be migrated and the second synchronization data.

[0073] Figure 5 The flowchart of performing consistency detection on the second data to be migrated and the second synchronization data in the data migration method according to an embodiment of the present disclosure is schematically shown.

[0074] like Figure 5 As shown, the method 500 includes steps S501 to S503.

[0075] Step S501: obtaining a data log of the second data to be migrated, performing data conversion, and generating data in a first lightweight exchange format.

[0076] Step S502: Acquire the data log of the second synchronization data, perform data conversion, and generate second lightweight exchange format data.

[0077] Step S503: performing consistency check on the table name, primary key, data type and field value of the first lightweight interchange format data and the second lightweight interchange format data.

[0078] Since the timestamps in the data log of the second synchronized data and the data log of the second data to be migrated are update timestamps and are different, the table name, primary key, data type, and field value are obtained for consistency check.

[0079] By lightweighting logs and checking consistency across table names, primary keys, data types, and field values, lightweight data can improve verification speed and conserve computer resources. Furthermore, checking based on table names, primary keys, data types, and field values ​​increases metadata detection accuracy, ultimately improving detection reliability.

[0080] Return to reference Figure 2 In step S207, if the detection is consistent, the first synchronization data and the second synchronization data are merged to generate target synchronization data, and the target synchronization data is stored in the target server to complete the data migration.

[0081] Figure 6 The flowchart of merging first synchronization data and second synchronization data to generate target synchronization data in the data migration method according to an embodiment of the present disclosure is schematically shown.

[0082] like Figure 6 As shown, the method 600 includes steps S601 to S604.

[0083] Step S601: Merge the first synchronization data and the second synchronization data to generate third synchronization data.

[0084] In step S602, all data in the first database are acquired, and consistency detection is performed on all data in the first database and the third synchronization data.

[0085] Step S603: If the detection is consistent, the third synchronization data is updated based on the incremental data of the first database to generate fourth synchronization data.

[0086] By updating the third synchronization data with the incremental data of the first database and generating the fourth synchronization data, the data change timestamp in the source database can be obtained, thereby improving the accuracy of data synchronization.

[0087] Step S604: merge the first synchronization data and the fourth synchronization data to generate target synchronization data.

[0088] The reliability of data migration is further improved by performing consistency check on all data in the first database and the third synchronization data, and the third synchronization data is updated based on the incremental data of the first database to generate the fourth synchronization data that can synchronize the timestamp of the incremental data, thereby further improving the accuracy and reliability of data migration.

[0089] Figure 7 The flowchart of performing consistency check on all data in the first database and the third synchronization data in the data migration method according to an embodiment of the present disclosure is schematically shown.

[0090] like Figure 7 As shown, the method 700 includes steps S701 to S704.

[0091] Step S701: Acquire the incremental data log in all the data in the first database and the log of the second synchronization data in the third synchronization data.

[0092] Step S702: Delete the heartbeat log and timestamp of the incremental data log in all the data in the first database to generate a first log.

[0093] Step S703: Delete the heartbeat log and timestamp of the log of the second synchronization data in the third synchronization data to generate a second log.

[0094] Step S704: Perform consistency check on the first log and the second log.

[0095] By deleting the heartbeat log and timestamp to eliminate the interference of non-core factors, the reliability of consistency detection is improved, while the detection efficiency is improved and computer and network resources are saved.

[0096] In addition, the heartbeat log in the first log and the heartbeat log in the second log can also be compared to verify the consistency of the first log and the second log, specifically including: if the heartbeat log in the first log and the heartbeat log in the second log are consistent, then the first log and the second log are consistent. If the heartbeat log in the first log and the heartbeat log in the second log are inconsistent, then if the time difference between the first log and the second log and the current time is within the first time configured by the prefabricated parameters, and there are no other incremental logs other than the mentality log within the second time configured by the prefabricated parameters, then the first log and the second log are consistent. Otherwise, the first log and the second log are inconsistent.

[0097] Figure 8 The flowchart of a specific embodiment of the data migration method according to the embodiment of the present disclosure is schematically shown.

[0098] like Figure 8 As described, this embodiment 800 includes: a transaction scheduling unit 801, a transaction execution unit 802, a source database master database 803, a source database backup database 804, a target database 805, a cross-database data replication unit 806, a full data verification unit 807, a log parsing unit 808, an incremental data storage unit 809 and an incremental data verification unit 810.

[0099] The transaction scheduling unit 801 is used to schedule business transactions to run between the source database 803 and the target database 805 cluster; perform second-level transaction interception and caching during switching to ensure that a single cluster has exclusive update permissions; and send a pre-switch consistency confirmation instruction to the incremental data verification unit 810.

[0100] The transaction execution unit 802 is responsible for providing a transaction execution environment, which is generally an application container or server.

[0101] The source database master database 803 is the source database master database, which is provided to the business system as the main data source before the switch.

[0102] The source database standby database 804 is a database product that uses the same source database as the primary database. Data synchronization between the primary and standby databases is generally achieved using the product's own replication technology. The data and logs of the primary and standby databases are completely consistent (including record update timestamps). Before the cross-database data replication unit 806 starts replication, data synchronization between the source database primary database 803 and the source database standby database 804 is disconnected, and the data in the source database standby database 804 is temporarily no longer changed.

[0103] The target database 805 and the source database are heterogeneous databases.

[0104] The cross-database data replication unit 806 implements data replication from the source database to the heterogeneous database, including full data replication and incremental data replication. Full replication is achieved through source database query + target database addition, incremental parsing of source-side logs, and application of add, update, modify, and delete statements in the target database. Therefore, the update timestamps in the incremental logs of the source and target databases are different. The update timestamp recorded in the source database standby database is the update commit timestamp in the source database master database, and the update timestamp in the target database log is the update commit timestamp executed in the target database.

[0105] The full data verification unit 807, after the cross-database data replication unit 806 completes the data replication, can perform a static comparison with the target database 805 because the data in the source database standby database 804 has not changed for the time being. Specifically, the data of the table with the same table name is divided into N intervals according to the primary key range, and is retrieved from the source database standby database 804 and the target database 805. Multi-threaded concurrent verification is used. If inconsistency is found, an exception log is recorded and an alarm is issued to notify the user.

[0106] The log parsing unit 808 parses the logs of the source database standby database 804 and the target database 805 to generate data in a lightweight exchange format. The content includes the database name + table name + primary key + update timestamp + update type + field values ​​before and after the record change. The same change of the same record in the source database standby database 804 and the target database 805 is completely consistent except for the update timestamp.

[0107] The incremental data storage unit 809 stores the data parsed by the log parsing unit 808 by classifying them by database name + table name and storing them in the order of update time.

[0108] The incremental data verification unit 810 reads the records of the source database and the target database in the incremental data storage unit 809, and processes them in parallel by table. Specifically, for a table, a certain number (such as 1,000) records of the source database and the target database are obtained in batches according to the update time sequence. After excluding the heartbeat records, they are compared. If they are consistent (other values ​​except the update timestamp are the same, including the primary key, update type and values ​​before and after the update), they are discarded. If no consistent version is found, they are merged into the next batch for comparison; if no consistent version is found in the batch with inconsistent versions exceeding a certain threshold, the verification difference table is written and an alarm is issued.

[0109] In addition, after receiving the instruction issued by the transaction scheduling unit 801 to confirm whether the incremental data is consistent before switching the database, if the difference between the incremental logs (including heartbeat logs) of the source and target databases and the current time is within 1 second (parameter configuration), and there are no new incremental logs except the heartbeat logs within 2 seconds, the incremental data of the source and target databases are completely consistent or the difference record is marked as eliminated, the transaction scheduling unit 801 is returned with a data comparison consistency.

[0110] The working principle of this embodiment 800 is as follows: the data replication between the source database master database 804 and the source database standby database 805 is disconnected so that the source database standby database data 804 no longer changes, and then the full data replication from the source database standby database 804 to the target database 805 is completed through the cross-database data replication unit 806. The full data verification unit 807 retrieves data from the source database standby database 804 and the target database 805, and uses multi-threaded concurrent verification. If inconsistencies are found, an abnormal log is recorded and an alarm is issued to notify the user. After the static comparison of the full data is completed, the data replication from the source database master database 803 to the source database standby database 804 is enabled, and the incremental data replication from the source database standby database 804 to the target database 805 is enabled through the cross-database data replication unit 806. The incremental data of the source database standby database 804 and the target database 805 is synchronized to the incremental data storage unit 809 through the log parsing unit 808. Incremental data verification is performed by the incremental data verification unit 810. Records from the source and target databases in the incremental data storage unit 809 are read and processed in parallel by table. Specifically, for a table, a certain number (e.g., 1,000) of records from the source and target databases are retrieved in batches, chronologically, excluding heartbeat records. These records are then compared. If they are consistent (other values ​​except the update timestamp are identical, including the primary key, update type, and pre- and post-update values), they are discarded. If no consistent versions are found, they are merged into the next batch for comparison. If no consistent versions are found in the batches with inconsistent versions exceeding a certain threshold, a verification difference table is written and an alarm is issued. The switch between the source database master 803 and the target database 805 begins. The transaction scheduling unit 801 begins intercepting cached transactions and sends instructions to the incremental data verification unit 810 for pre-switch verification. Within a certain timeframe, the incremental data verification unit 810 verifies that the difference between the source and target database incremental logs (including heartbeat logs) and the current time is within one second (the maximum time stamp is 103, from the heartbeat of the source data master database to the source data backup database 804, to the target database 805, and then through the log parsing unit 808 to the incremental data storage unit 805). Furthermore, no new incremental logs other than the heartbeat logs have been added within two seconds. The incremental data on the source and target databases is completely consistent or the difference records are marked as eliminated, and the data comparison is returned as consistent. The transaction invocation unit 810 switches the intercepted transaction to the transaction execution unit 802 corresponding to the target database 805, completing the database switch.

[0111] This embodiment achieves heterogeneous database data comparison and switching through the cooperation of a transaction scheduling unit, a cross-database data replication unit, a full data verification unit, and an incremental data verification unit, providing uninterrupted external services. It achieves switching within seconds through dynamic standby databases and dual-log comparison. It adapts to multiple database types based on standardized log parsing technology. It reduces data risk through full and incremental dual-stage verification. Because related operations are performed on the source database standby database, there is no performance impact on the primary database. The same solution can be used on the target database to build a standby database, achieving uninterrupted service failover, significantly reducing the business risk of heterogeneous database migration.

[0112] Figure 9 The structural block diagram of the data migration device according to an embodiment of the present disclosure is schematically shown.

[0113] like Figure 9 As shown, the apparatus 900 includes: a first generating module 901 , a second generating module 902 , a first detecting module 903 , a third generating module 904 , a fourth generating module 905 , a second detecting module 906 and a fifth generating module 907 .

[0114] The first generating module 901 is configured to terminate the data backup of the first database by the second database, obtain all data in the second database, and generate first data to be migrated. In one embodiment, the first generating module 901 may be configured to execute step S201 described above, which will not be described in detail here.

[0115] The second generating module 902 is configured to synchronize the first data to be migrated to the target database to generate first synchronization data. In one embodiment, the second generating module 902 can be configured to execute step S202 described above, which will not be described in detail here.

[0116] The first detection module 903 is configured to perform consistency detection on the first data to be migrated and the first synchronization data. In one embodiment, the first detection module 903 may be configured to execute step S203 described above.

[0117] The first detection module 903 includes: a first acquisition module, a third detection module, a sixth generation module and a fourth detection module.

[0118] The first acquisition module is configured to acquire a plurality of first data tables of the first data to be migrated and a plurality of second data tables of the first synchronization data. In one embodiment, the first acquisition module may be configured to execute step S301 described above, which will not be described in detail here.

[0119] The third detection module is configured to detect whether the table names of the plurality of first data tables and the plurality of second data tables are all the same. In one embodiment, the third detection module may be configured to execute step S302 described above, which will not be described in detail here.

[0120] The sixth generation module is configured to, if the table names of the plurality of first data tables and the plurality of second data tables are all the same, perform primary key range segmentation on the data of the first data tables and the data of the second data tables having the same table names, thereby generating n intervals of data for the data of the first data tables and n intervals of data for the data of the second data tables, where n is an integer greater than 1. In one embodiment, the sixth generation module may be configured to execute step S303 described above and will not be further described herein.

[0121] The fourth detection module is configured to perform a one-to-one consistency detection on the n intervals of data in the first data table and the n intervals of data in the second data table. In one embodiment, the fourth detection module may be configured to execute step S304 described above.

[0122] The fourth detection module includes: a first calculation module and a fifth detection module.

[0123] The first calculation module is configured to calculate hash values ​​of corresponding data in the n intervals of data in the first data table and the n intervals of data in the second data table using multiple threads. In one embodiment, the first calculation module can be configured to execute step S401 described above, which will not be described in detail here.

[0124] A fifth detection module is configured to perform consistency detection based on the hash values ​​of the corresponding data in the n intervals of data in the first data table and the n intervals of data in the second data table. In one embodiment, the fifth detection module can be configured to execute step S402 described above, which will not be described in detail here.

[0125] The third generation module 904 is configured to, if a match is detected, initiate a data backup of the first database by the second database, obtain incremental data in the second database, and generate second data to be migrated. The incremental data in the second database is the incremental data in the first database backed up by the second database after the data backup of the first database by the second database is terminated. In one embodiment, the third generation module 904 can be configured to execute step S204 described above and will not be further described here.

[0126] The fourth generating module 905 is configured to synchronize the second data to be migrated to the target database to generate second synchronized data. In one embodiment, the fourth generating module 905 may be configured to execute step S205 described above, which will not be described in detail here.

[0127] The second detection module 906 is configured to perform consistency detection on the second data to be migrated and the second synchronization data. In one embodiment, the second detection module 906 may be configured to execute step S206 described above.

[0128] The second detection module 906 includes a seventh generation module, an eighth generation module and a sixth detection module.

[0129] The seventh generating module is configured to obtain the data log of the second data to be migrated, perform data conversion, and generate data in the first lightweight exchange format. In one embodiment, the seventh generating module may be configured to execute step S501 described above, which will not be described in detail here.

[0130] The eighth generation module is configured to obtain the data log of the second synchronization data, perform data conversion, and generate second lightweight exchange format data. In one embodiment, the eighth generation module may be configured to execute step S502 described above, which will not be described in detail here.

[0131] The sixth detection module is configured to perform consistency detection on the table name, primary key, data type, and field value of the first lightweight exchange format data and the second lightweight exchange format data. In one embodiment, the sixth detection module may be configured to execute step S503 described above, which will not be described in detail here.

[0132] The fifth generation module 907 is configured to merge the first synchronization data and the second synchronization data to generate target synchronization data if the detected data are consistent, and store the target synchronization data in the target server to complete the data migration. In one embodiment, the fifth generation module 907 can be configured to execute step S207 described above.

[0133] The fifth generating module 907 includes: a ninth generating module, a seventh detecting module, a tenth generating module and an eleventh generating module.

[0134] The ninth generating module is configured to merge the first synchronization data and the second synchronization data to generate third synchronization data. In one embodiment, the ninth generating module may be configured to execute step S601 described above, which will not be described in detail here.

[0135] The seventh detection module is configured to obtain all data in the first database and perform consistency detection on all data in the first database and the third synchronization data. In one embodiment, the seventh detection module may be configured to execute step S602 described above.

[0136] The seventh detection module includes: a second acquisition module, a twelfth generation module, a thirteenth generation module and a seventh detection module.

[0137] The second acquisition module is used to acquire the incremental data logs from all the data in the first database and the logs of the second synchronization data from the third synchronization data. In one embodiment, the second acquisition module can be used to execute step S701 described above, which will not be described in detail here.

[0138] The twelfth generation module is configured to delete the heartbeat log and timestamp of the incremental data log in all data of the first database to generate the first log. In one embodiment, the twelfth generation module may be configured to execute step S702 described above, which will not be described in detail here.

[0139] The thirteenth generation module is configured to delete the heartbeat log and timestamp of the log of the second synchronization data in the third synchronization data to generate a second log. In one embodiment, the thirteenth generation module may be configured to execute step S703 described above, which will not be described in detail here.

[0140] The eighth detection module is configured to perform consistency detection on the first log and the second log. In one embodiment, the eighth detection module may be configured to execute step S704 described above, which will not be described in detail here.

[0141] The tenth generation module is configured to update the third synchronization data based on the incremental data of the first database to generate the fourth synchronization data if the detection is consistent. In one embodiment, the tenth generation module can be configured to execute step S603 described above, which will not be described in detail here.

[0142] The eleventh generating module is configured to merge the first synchronization data and the fourth synchronization data to generate target synchronization data. In one embodiment, the eleventh generating module may be configured to execute step S604 described above, which will not be described in detail here.

[0143] According to embodiments of the present disclosure, any multiple of the first generation module 901, the second generation module 902, the first detection module 903, the third generation module 904, the fourth generation module 905, the second detection module 906, and the fifth generation module 907 can be combined into a single module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in a single module. According to embodiments of the present disclosure, at least one of the first generation module 901, the second generation module 902, the first detection module 903, the third generation module 904, the fourth generation module 905, the second detection module 906, and the fifth generation module 907 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of any of these. Alternatively, at least one of the first generation module 901, the second generation module 902, the first detection module 903, the third generation module 904, the fourth generation module 905, the second detection module 906 and the fifth generation module 907 can be at least partially implemented as a computer program module, which can perform the corresponding function when it is run.

[0144] Figure 10 A block diagram of an electronic device suitable for implementing a data migration method according to an embodiment of the present disclosure is schematically shown.

[0145] like Figure 10 As shown, the electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0146] Various programs and data required for the operation of the electronic device 1000 are stored in the RAM 1003. The processor 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. The processor 1001 performs various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 1002 and / or the RAM 1003. It should be noted that the programs may also be stored in one or more memories other than the ROM 1002 and the RAM 1003. The processor 1001 may also perform various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0147] According to an embodiment of the present disclosure, electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to bus 1004. Electronic device 1000 may also include one or more of the following components connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 1008 including a hard disk; and a communication section 1009 including a network interface card such as a LAN card or modem. Communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 1010 as needed, so that computer programs read from the removable media can be installed into storage section 1008 as needed.

[0148] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0149] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include ROM 1002 and / or RAM 1003 described above and / or one or more memories other than ROM 1002 and RAM 1003.

[0150] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data migration method provided by the embodiments of the present disclosure.

[0151] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the processor 1001 executes the computer program. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0152] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 1009, and / or installed from the removable medium 1011. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0153] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0154] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0156] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0157] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data migration method, characterized in that: The method includes: terminating the data backup of the first database by the second database, obtaining all the data in the second database, and generating first data to be migrated; Synchronizing the first data to be migrated to the target database to generate first synchronized data; performing a consistency check on the first data to be migrated and the first synchronization data; If the detection is consistent, start backing up the data of the first database with the second database, obtain the incremental data in the second database, and generate the second data to be migrated, wherein the incremental data in the second database is the incremental data in the first database backed up by the second database after the data backup of the first database by the second database is terminated; Synchronizing the second data to be migrated into the target database to generate second synchronized data; performing a consistency check on the second data to be migrated and the second synchronization data; and If the detection is consistent, the first synchronization data and the second synchronization data are merged to generate target synchronization data, and the target synchronization data is stored in the target server to complete the data migration.

2. The method according to claim 1, characterized in that Performing consistency check on the first data to be migrated and the first synchronization data includes: Acquire multiple first data tables of the first data to be migrated and multiple second data tables of the first synchronized data; detecting whether the table names of the plurality of first data tables and the plurality of second data tables are all the same; If the table names of the plurality of first data tables and the plurality of second data tables are all the same, performing primary key range segmentation on the data of the first data tables and the data of the second data tables having the same table names, generating n interval data of the data of the first data tables and n interval data of the data of the second data tables, where n is an integer and n is greater than 1; and A one-to-one consistency check is performed on the n interval data of the data in the first data table and the n interval data of the data in the second data table.

3. The method according to claim 2, characterized in that Performing one-to-one consistency detection on n intervals of data in the first data table and n intervals of data in the second data table, including: Calculating hash values ​​of one-to-one corresponding data in n intervals of data in the first data table and n intervals of data in the second data table using multiple threads; and Consistency detection is performed using hash values ​​of one-to-one corresponding data in the n interval data of the data in the first data table and the n interval data of the data in the second data table.

4. The method according to claim 1, wherein Performing consistency check on the second data to be migrated and the second synchronization data includes: Obtaining a data log of the second data to be migrated, performing data conversion, and generating data in a first lightweight exchange format; Obtaining a data log of the second synchronization data, performing data conversion, and generating second lightweight exchange format data; and A consistency check is performed on the table name, primary key, data type, and field value of the first lightweight interchange format data and the second lightweight interchange format data.

5. The method according to any one of claims 1 to 4, characterized in that Merging the first synchronization data and the second synchronization data to generate target synchronization data includes: Merging the first synchronization data and the second synchronization data to generate third synchronization data; then obtaining all the data in the first database, and performing consistency check on all the data in the first database and the third synchronization data; If the detection is consistent, the third synchronization data is updated based on the incremental data of the first database to generate the fourth synchronization data, and The first synchronization data and the fourth synchronization data are merged to generate target synchronization data.

6. The method according to claim 5, characterized in that Performing consistency check on all data in the first database and the third synchronization data includes: Obtaining incremental data logs from all data in the first database and logs of second synchronization data from the third synchronization data; Deleting the heartbeat log and timestamp of the incremental data log in all data of the first database to generate a first log; Deleting the heartbeat log and timestamp of the log of the second synchronization data in the third synchronization data to generate a second log; and Perform a consistency check on the first log and the second log.

7. A data migration device, characterized in that: The device includes: A first generating module is configured to stop the second database from backing up data of the first database, obtain all data in the second database, and generate first data to be migrated; A second generating module is configured to synchronize the first data to be migrated into a target database to generate first synchronization data; a first detection module, configured to perform consistency detection on the first data to be migrated and the first synchronization data; a third generating module, configured to, if the detected data are consistent, start backing up the first database with the second database, obtain incremental data in the second database, and generate second data to be migrated, wherein the incremental data in the second database is the incremental data in the first database backed up by the second database after stopping the data backing up of the first database by the second database; a fourth generating module, configured to synchronize the second data to be migrated into a target database to generate second synchronized data; a second detection module, configured to perform consistency detection on the second data to be migrated and the second synchronization data; and The fifth generation module is configured to merge the first synchronization data and the second synchronization data to generate target synchronization data if the detected data are consistent, and store the target synchronization data in a target server to complete data migration.

8. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, The method further comprises the step of executing the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images