Data bidirectional migration method, device, equipment, medium and program product

Abstract

This disclosure provides a bidirectional data migration method applicable to the field of information security technology. The method includes: monitoring whether log information in a source database has changed; if the log information has changed, acquiring changed data from different data tables in the source database, wherein the changed data includes flag fields; changing the flag fields in the changed data; converting the changed data into a change statement, wherein the change statement is executable at least in the target database; and if the flag fields in the change statement are not default values, performing a migration operation with the target database based on the change statement. This disclosure also provides a bidirectional data migration apparatus, device, storage medium, and program product.

Description

Technical Field

[0001] This disclosure relates to the field of information security technology, and more specifically to a method, apparatus, device, medium, and program product for bidirectional data migration. Background Technology

[0002] In the field of data migration, the main approach is one-way migration, which means migrating from the source database and immediately switching to the target database. However, if the target database experiences functional problems or a cluster of servers crashes, it becomes impossible to back up incremental data or switch back to the source database.

[0003] Furthermore, during the migration process, if the target database encounters an anomaly or the migration tool malfunctions, data loss or duplicate migrations can easily occur, making it difficult to guarantee the synchronization of data between the two databases. Summary of the Invention

[0004] In view of the above problems, this disclosure provides methods, apparatus, devices, media and program products for bidirectional data migration that improve migration security and can be performed without downtime.

[0005] According to a first aspect of this disclosure, a bidirectional data migration method is provided, comprising: monitoring whether log information in a source database has changed; if the log information has changed, acquiring changed data under different data tables in the source database, wherein the changed data includes a flag field; changing the flag field in the changed data; converting the changed data into a change statement, wherein the change statement is executable at least in the target database; and if the flag field of the change statement is a non-default value, performing a migration operation with the target database based on the change statement.

[0006] According to embodiments of this disclosure, obtaining the changed data under different data tables in the source database includes: obtaining the data tables involved in the changed data; creating topics, segmented channels, and consumer models based on message middleware, wherein each topic corresponds one-to-one with a data table, each consumer model corresponds one-to-one with a data table, and each topic has a one-to-many relationship with a segmented channel; storing the changed data in the topic according to the one-to-one correspondence between the topic and the data table; and retrieving the changed data in batches based on the consumer model.

[0007] According to an embodiment of this disclosure, wherein the batch includes a first batch, and after the migration operation with the target database is performed based on the change statement, the method further includes: checking whether multiple topics under the first batch have been successfully migrated based on the returned migration result of the segmentation channel, wherein the multiple topics include a first topic, and if any migration result under the first topic is a failure, then the first topic is determined to have failed to migrate; if the first topic has been successfully migrated, recording a first offset of the last migration result of the first topic, incrementing the first offset by one, and submitting it; and if the first topic has failed to migrate, recording a second offset of the failed migration result in the segmentation channel, and submitting the second offset.

[0008] According to an embodiment of this disclosure, the segmentation channel includes at least a first channel. The step of checking whether multiple topics in the first batch have been successfully migrated based on the migration results of the returned segmentation channel includes: for the first channel, traversing multiple migration results and determining whether the migration result has failed; and if it is determined that the migration result has failed after traversing multiple migration results, determining that the current topic in which the migration result is located has failed, stopping the traversal of multiple segmentation channels in the current topic and entering the next topic.

[0009] According to an embodiment of this disclosure, the change statement is executable in the source database, and after converting the change data into a change statement, the method further includes: backing up the change statement to the source database.

[0010] According to embodiments of this disclosure, the source database includes an Oracle database and a MySQL database, and the target database includes an Oracle database and a MySQL database. The step of obtaining changed data from different tables in the source database when the log information changes includes: for the Oracle database, parsing the materialized view log to obtain the binary code of the changed object; querying the binary code of the changed object to locate the primary key field, ordinary field, and value of the changed data; packaging and encapsulating the primary key field, the ordinary field, and the value to obtain the changed data; for the MySQL database, parsing the log file to obtain the timestamp of the change time node; querying the SQL statement based on the timestamp; and packaging the SQL statement to obtain the changed data.

[0011] A second aspect of this disclosure provides a bidirectional data migration apparatus, comprising: a log monitoring module for monitoring whether log information in a source database has changed; a changed data acquisition module for acquiring changed data under different data tables in the source database when the log information has changed, wherein the changed data includes a flag field; a flag changing module for changing the flag field in the changed data; a statement conversion module for converting the changed data into a change statement, wherein the change statement is executable at least in the target database; and a migration module for performing a migration operation with the target database based on the change statement when the flag field of the change statement is not a default value.

[0012] According to embodiments of this disclosure, the change data acquisition module is used to acquire data tables related to the change data; create topics, segmented channels, and consumer models based on message middleware, wherein each topic corresponds one-to-one with a data table, each consumer model corresponds one-to-one with a data table, and each topic has a one-to-many relationship with a segmented channel; store the change data in the topic according to the one-to-one correspondence between the topic and the data table; and retrieve the change data in batches based on the consumer model.

[0013] According to embodiments of this disclosure, the batch includes a first batch, and the device further includes a result return module for checking whether multiple topics under the first batch have been successfully migrated based on the returned migration results of the segmentation channel, wherein the multiple topics include a first topic, and if any migration result under the first topic is a failure, the first topic is determined to have failed to migrate; if the first topic is successfully migrated, a first offset of the last migration result of the first topic is recorded, and the first offset is incremented by one before submission; and if the first topic fails to migrate, a second offset of the failed migration result in the segmentation channel is recorded, and the second offset is submitted.

[0014] According to an embodiment of this disclosure, the segmentation channel includes at least a first channel, and the result return module is configured to, for the first channel, traverse multiple migration results and determine whether the migration result has failed; and, if the migration result has failed after traversing multiple migration results, determine that the current topic in which the migration result is located has failed, stop traversing multiple segmentation channels in the current topic and proceed to the next topic.

[0015] According to an embodiment of this disclosure, the apparatus further includes a backup module for backing up the change statements to the source database.

[0016] According to embodiments of this disclosure, the source database includes an Oracle database and a MySQL database, and the target database includes an Oracle database and a MySQL database. The change data acquisition module is used to: for the Oracle database, parse the materialized view log to obtain the binary code of the changed object; query the binary code of the changed object to locate the primary key field, ordinary field, and value of the change data; and package the primary key field, ordinary field, and value to obtain the change data. For the MySQL database, parse the log file to obtain the change time node timestamp; query the SQL statement based on the time node timestamp; and package the SQL statement to obtain the change data.

[0017] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the aforementioned bidirectional data migration method.

[0018] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the aforementioned bidirectional data migration method.

[0019] A fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described bidirectional data migration method.

[0020] In the embodiments disclosed herein, the following beneficial effects can be achieved: 1. By identifying the data source of the changed data, bidirectional synchronous migration is achieved between the source database and the target database that generates the business change, ensuring the real-time nature of the bidirectional data migration; 2. By setting a flag bit, the status of the data change is determined, ensuring the effective migration of the data and avoiding the cyclical bidirectional repeated migration of the same changed data in different databases in the bidirectional migration scenario. Attached Figure Description

[0021] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0022] Figure 1 This diagram illustrates an application scenario of the bidirectional data migration method according to embodiments of the present disclosure.

[0023] Figure 2 A flowchart illustrating a bidirectional data migration method according to an embodiment of the present disclosure is shown schematically.

[0024] Figure 3A flowchart illustrating a change data acquisition method according to an embodiment of the present disclosure is shown schematically.

[0025] Figure 4 A flowchart illustrating a bidirectional data migration method according to an embodiment of the present disclosure is shown schematically.

[0026] Figure 5 A flowchart illustrating a migration result determination method according to an embodiment of the present disclosure is shown schematically.

[0027] Figure 6 A schematic diagram illustrating the entire process of a bidirectional data migration method according to an embodiment of the present disclosure is provided.

[0028] Figure 7 A schematic block diagram of a bidirectional data migration apparatus according to an embodiment of the present disclosure is shown; and

[0029] Figure 8 A block diagram of an electronic device suitable for implementing a bidirectional data migration method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0030] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0031] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0032] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0033] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0034] Before detailing the embodiments of this disclosure, the key technical terms involved in the embodiments of this disclosure will be explained one by one, as follows:

[0035] Batch: Used to improve the efficiency of message middleware by writing messages to the message middleware in batches.

[0036] Subject: Messages are categorized by subject, and a subject can be divided into several sub-channels.

[0037] Offset: Represents the index of the message that the message middleware periodically retrieves, and the specific value it moves relative to the previous index position.

[0038] Changes to one database can lead to duplicate writes of incremental data to both the source and target databases, creating a loop and causing repeated writes. Additionally, during data migration, a crash in either the source or target database, or an unexpected stop in the migration process, can result in repeated DML operations on incremental data upon restart. When these situations occur during incremental data migration, it's necessary to preprocess the incremental data to be migrated, locate the error in the data, discard the erroneous data, and perform a re-migration to prevent incorrect data recording and subsequent data desynchronization between the source and target databases.

[0039] To address the technical problems existing in the prior art, embodiments of this disclosure provide a bidirectional data migration method, which involves monitoring whether log information in a source database has changed; if the log information has changed, acquiring changed data from different data tables in the source database, wherein the changed data includes a flag field; changing the flag field in the changed data; converting the changed data into a change statement, wherein the change statement is executable at least in the target database; and if the flag field of the change statement is not a default value, performing a migration operation with the target database based on the change statement.

[0040] In the embodiments disclosed herein, the following beneficial effects can be achieved: 1. By identifying the data source of the changed data, bidirectional synchronous migration is achieved between the source database and the target database that generates the business change, ensuring the real-time nature of the bidirectional data migration; 2. By setting a flag bit, the status of the data change is determined, ensuring the effective migration of the data and avoiding the cyclical bidirectional repeated migration of the same changed data in different databases in the bidirectional migration scenario.

[0041] Figure 1 The diagram illustrates an application scenario of the bidirectional data migration method according to an embodiment of the present disclosure.

[0042] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0043] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0044] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0045] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0046] It should be noted that the bidirectional data migration method provided in this embodiment can generally be executed by server 105. Correspondingly, the bidirectional data migration device provided in this embodiment can generally be located in server 105. The bidirectional data migration method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the bidirectional data migration device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0047] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0048] The following will be based on Figure 1 The described scene, through Figures 2-6 The bidirectional data migration method of the disclosed embodiments will be described in detail.

[0049] Figure 2 A flowchart illustrating a bidirectional data migration method according to an embodiment of the present disclosure is shown schematically.

[0050] like Figure 2 As shown, the bidirectional data migration method of this embodiment includes operations S210 to S250, and the bidirectional data migration method can be executed by server 105.

[0051] In operation S210, monitor whether the log information in the source database has changed.

[0052] Specifically, the embodiments of this disclosure can be applied to achieve bidirectional data migration between the source and target databases without downtime. The source and target databases can be databases in the same environment, for example, both can be Oracle databases, or they can be databases connected via MySQL. Alternatively, the source and target databases can be databases in different environments, for example, one can be an Oracle database and the other a MySQL database, or the other a MySQL database and the other an Oracle database.

[0053] Specifically, the aforementioned bidirectional migration includes both incremental and decremental data migration. For example, when adding data to the source database, the incremental data needs to be migrated to the target database. Similarly, when deleting data from the source database, the source database in operation S210 specifically refers to the origin of the data modification; it is relative, not a specific database. For instance, when an Oracle database and a MySQL database are each other's source and target databases, when data changes in the Oracle database, the Oracle database becomes the source database, and when data changes in the MySQL database, the Oracle database becomes the source database.

[0054] In operation S220, when the log information changes, the changed data under different data tables in the source database is obtained, wherein the changed data includes a flag field.

[0055] According to embodiments of this disclosure, the source database includes an Oracle database and a MySQL database, and the target database includes an Oracle database and a MySQL database. The step of obtaining changed data from different tables in the source database when the log information changes includes: for the Oracle database, parsing the materialized view log to obtain the binary code of the changed object; querying the binary code of the changed object to locate the primary key field, ordinary field, and value of the changed data; packaging and encapsulating the primary key field, the ordinary field, and the value to obtain the changed data; for the MySQL database, parsing the log file to obtain the timestamp of the change time node; querying the SQL statement based on the timestamp; and packaging the SQL statement to obtain the changed data.

[0056] Specifically, taking incremental migration scenarios as examples, different database environments are used:

[0057] For Oracle databases, before enabling incremental data migration, materialized views and materialized view logs are created based on the primary key fields of the tables to be migrated. When data changes occur in the Oracle database tables, the materialized view log file information is parsed. By querying the binary code of the changed objects, the primary key fields, ordinary fields, and corresponding values ​​of the incremental data are located, and the above information is re-encapsulated into JSON format for recording.

[0058] For MySQL databases, when data changes occur in MySQL database tables, monitor the bin log file (binary log file), and query and retrieve the SQL statement corresponding to the changed data based on the timestamp of the change time point.

[0059] In operation S230, the field of the flag bit in the changed data is modified.

[0060] Specifically, if incremental data exists in both the Oracle and MySQL databases within a certain period, the corresponding default flag values ​​will be changed based on the incremental data in both databases. The changed data regarding the flag field values ​​will then be encapsulated.

[0061] In operation S240, the changed data is converted into a change statement, wherein the change statement is executable at least in the target database.

[0062] According to an embodiment of this disclosure, the change statement is executable in the source database, and after converting the change data into a change statement, the method further includes: backing up the change statement to the source database.

[0063] After judging the flag bit, bidirectional writing is performed to the target database and the source database to complete the synchronous backup of data from two data sources and improve data security.

[0064] Specifically, it generates executable SQL statements for both the source and target databases. For example, this SQL statement could be an UPDATE statement. Researchers have found that SQL statements are usable in both Oracle and MySQL databases; therefore, the aforementioned change statements can be implemented using SQL.

[0065] In operation S250, if the field of the flag bit of the change statement is not a default value, a migration operation with the target database is performed based on the change statement.

[0066] Specifically, the Oracle and MySQL databases are respectively configured to check the written SQL statements and parse the flag field values. If the flag field value is determined to be modified, then by default, the modified statements generated in the Oracle database do not need to be migrated back to the Oracle database, and the modified statements generated in the MySQL database do not need to be migrated back to the MySQL database. A persistence layer framework is used for database writes or other business processing. Of course, in scenarios requiring backups, the SQL statements can be written to both the Oracle and MySQL databases simultaneously for backup. This allows for rapid failover if one database experiences an unknown error that causes a system crash.

[0067] In the embodiments disclosed herein, the following beneficial effects can be achieved: 1. By identifying the data source of the changed data, bidirectional synchronous migration is achieved between the source database and the target database that generates the business change, ensuring the real-time nature of the bidirectional data migration; 2. By setting a flag bit, the status of the data change is determined, ensuring the effective migration of the data and avoiding the cyclical bidirectional repeated migration of the same changed data in different databases in the bidirectional migration scenario.

[0068] Since the source and target databases are independent of each other and data is written and written frequently, a message middleware can be set up to handle the frequent bidirectional synchronization and migration operations between the two databases.

[0069] Figure 3 A flowchart illustrating a change data acquisition method according to an embodiment of this disclosure is shown schematically.

[0070] like Figure 3 As shown, the change data acquisition method of this embodiment includes operations S310 to S340, which can at least partially perform the above-mentioned operation S220.

[0071] In operation S310, the data table involved in the changed data is obtained.

[0072] The database contains multiple tables, and operations performed on different tables will result in changes to the data. In other words, changes obtained through log changes can pinpoint the table in which they occurred.

[0073] In operation S320, topics, segmented channels, and consumer models are created based on the message middleware. The topics correspond one-to-one with the data tables, the consumer models correspond one-to-one with the data tables, and the topics and segmented channels have a one-to-many relationship.

[0074] It's understandable that a topic is a logical concept, and topics and data tables have a certain correspondence. Generally, according to business logic, topics and data tables have a one-to-one correspondence. For example, if a data table records data for "Business 1", then that table corresponds to the topic "Business 1".

[0075] In operation S330, the changed data is stored in the topic according to the one-to-one correspondence between the topic and the data table.

[0076] In operation S340, the change data is retrieved in batches based on the consumer model.

[0077] Specifically, based on the number of tables to be modified in the source and target databases, corresponding Topic topics and corresponding Partitions are created using message middleware. The re-encapsulation information corresponding to the incremental data in the source and target databases is sent to the corresponding Topic topics in the message middleware. A corresponding number of Consumer models are created based on the number of topics in the message middleware. Messages are pulled in batches at regular intervals (or in fixed quantities). Based on different Topic topics as keys, incremental data change information in the source and target database tables is distinguished to ensure that the information to be consumed in the Partitions under the same Topic is from the same data source database order table.

[0078] It is understandable that the segmented channel can be regarded as a channel, and the migration of the channel is serial. During the migration of a topic, a topic can realize the migration of characters according to the serial channels of multiple segmented channels.

[0079] In the embodiments of this disclosure, a message middleware can be used to achieve bidirectional synchronous migration between the source database and the target database that generate business changes, ensuring that business data can be exchanged between different databases in a decoupled manner.

[0080] Figure 4 A flowchart illustrating a bidirectional data migration method according to an embodiment of the present disclosure is shown schematically.

[0081] like Figure 4 As shown, the bidirectional data migration method of this embodiment includes operations S410 to S430, which are performed after operation S260.

[0082] According to embodiments of this disclosure, the batch includes the first batch.

[0083] It is understandable that "first batch" refers to any particular batch.

[0084] In operation S410, based on the returned migration results of the segmented channel, it is checked whether multiple topics under the first batch have been successfully migrated, wherein the multiple topics include the first topic. If any migration result under the first topic is a failure, the migration of the first topic is determined to have failed.

[0085] In operation S420, if the migration of the first topic is successful, the first offset of the last migration result of the first topic is recorded, and the first offset is incremented by one before submission.

[0086] In operation S430, if the first topic migration fails, the second offset of the migration result that failed in the segmentation channel is recorded and the second offset is submitted.

[0087] It is understandable that the first offset mentioned above corresponds to the offset in the case of success, and the second offset mentioned above corresponds to the offset in the case of failure.

[0088] Retrieve the persistent layer framework write results of all threads in the same batch. If network delays occur during data migration, or if the message middleware unexpectedly stops running due to actual production environment issues, use the Topic (which generally corresponds to a data table) as the distinguishing information to iterate through all returned results and monitor all returned information.

[0089] Specifically, after all the return values ​​of the partition segmentation channel messages under different topics in this batch have been traversed, the offset value of the last message in different topics in this batch is recorded. At the same time, the offset of the partition segmentation channel offset plus 1 is submitted under topics where no data import error occurred. For partitions under topics where data import error occurred, the offset value of the message in which the error occurred must be submitted.

[0090] Similarly, after each batch of data has been consumed, the offset of messages that may have caused errors during the data migration process can be monitored, recorded, and submitted to the message middleware, while simultaneously handling error reports. This ensures the orderliness and uniqueness of incremental data in the database.

[0091] Figure 5 A flowchart illustrating a migration result determination method according to an embodiment of the present disclosure is shown schematically.

[0092] like Figure 5 As shown, the migration result determination method of this embodiment includes operations S510 to S520, which can at least partially execute the above-mentioned operation S410.

[0093] If the first topic migration fails, record the second offset of the migration result that failed in the segmentation channel, and submit the second offset in operation S510. For the first channel, traverse multiple migration results and determine whether the migration result has failed.

[0094] In operation S520, if the migration result fails during the traversal of multiple migration results, the migration of the current topic in which the migration result is located is determined to have failed, the traversal of multiple segmentation channels in the current topic is stopped, and the next topic is entered.

[0095] Specifically, if a partition channel within a single Topic returns a false result and fails, the offset of the failed message is recorded, and the message loop for that topic is stopped before proceeding to the next topic. This effectively utilizes the partition channel's resources, maximizing its utilization.

[0096] Figure 6 A schematic diagram illustrating the complete flow of a bidirectional data migration method according to an embodiment of the present disclosure is provided.

[0097] like Figure 6 As shown, the bidirectional data migration method includes: parsing logs to obtain changed data when Oracle and / or MySQL database forms change; creating corresponding Oracle / MySQL topics; pulling messages from Oracle / MySQL to the topics; determining if a flag is at its default value; if so, modifying the flag and assembling an SQL statement to write to the MySQL / Oracle database; otherwise, determining if it is backup data, no write-back is needed, and writing to the database; determining whether the write was successful based on the return value; if so, recording the written data position; otherwise, recording the error position and committing the offset, while simultaneously interrupting the data migration.

[0098] Based on the above-described bidirectional data migration method, this disclosure also provides a bidirectional data migration apparatus. The following will be combined with... Figure 7 The device is described in detail.

[0099] Figure 7 A schematic block diagram of a bidirectional data migration apparatus according to an embodiment of the present disclosure is shown.

[0100] like Figure 7 As shown, the bidirectional data migration device 700 of this embodiment includes a log monitoring module 710, a change data acquisition module 720, a flag bit changing module 730, a statement conversion module 740, and a migration module 750.

[0101] The log monitoring module 710 is used to monitor whether the log information in the source database has changed. In one embodiment, the log monitoring module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0102] The change data acquisition module 720 is used to acquire change data from different data tables in the source database when the log information changes, wherein the change data includes a flag field. In one embodiment, the change data acquisition module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0103] The flag bit changing module 730 is used to change the flag bit field in the changed data. In one embodiment, the flag bit changing module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0104] The statement conversion module 740 is used to convert the changed data into change statements, wherein the change statements are executable at least in the target database. In one embodiment, the statement conversion module 740 can be used to perform the operation S240 described above, which will not be repeated here.

[0105] The migration module 750 is used to perform a migration operation with the target database based on the change statement when the field of the flag bit in the change statement is not a default value. In one embodiment, the migration module 750 can be used to perform the operation S250 described above, which will not be repeated here.

[0106] In the embodiments disclosed herein, the following beneficial effects can be achieved: 1. By identifying the data source of the changed data, bidirectional synchronous migration is achieved between the source database and the target database that generates the business change, ensuring the real-time nature of the bidirectional data migration; 2. By setting a flag bit, the status of the data change is determined, ensuring the effective migration of the data and avoiding the cyclical bidirectional repeated migration of the same changed data in different databases in the bidirectional migration scenario.

[0107] According to embodiments of this disclosure, the change data acquisition module is used to acquire data tables related to the change data; create topics, segmented channels, and consumer models based on message middleware, wherein each topic corresponds one-to-one with a data table, each consumer model corresponds one-to-one with a data table, and each topic has a one-to-many relationship with a segmented channel; store the change data in the topic according to the one-to-one correspondence between the topic and the data table; and retrieve the change data in batches based on the consumer model.

[0108] According to embodiments of this disclosure, the batch includes a first batch, and the device further includes a result return module for checking whether multiple topics under the first batch have been successfully migrated based on the returned migration results of the segmentation channel, wherein the multiple topics include a first topic, and if any migration result under the first topic is a failure, the first topic is determined to have failed to migrate; if the first topic is successfully migrated, a first offset of the last migration result of the first topic is recorded, and the first offset is incremented by one before submission; and if the first topic fails to migrate, a second offset of the failed migration result in the segmentation channel is recorded, and the second offset is submitted.

[0109] According to an embodiment of this disclosure, the segmentation channel includes at least a first channel, and the result return module is configured to, for the first channel, traverse multiple migration results and determine whether the migration result has failed; and, if the migration result has failed after traversing multiple migration results, determine that the current topic in which the migration result is located has failed, stop traversing multiple segmentation channels in the current topic and proceed to the next topic.

[0110] According to an embodiment of this disclosure, the apparatus further includes a backup module for backing up the change statements to the source database.

[0111] According to embodiments of this disclosure, the source database includes an Oracle database and a MySQL database, and the target database includes an Oracle database and a MySQL database. The change data acquisition module is used to: for the Oracle database, parse the materialized view log to obtain the binary code of the changed object; query the binary code of the changed object to locate the primary key field, ordinary field, and value of the change data; and package the primary key field, ordinary field, and value to obtain the change data. For the MySQL database, parse the log file to obtain the change time node timestamp; query the SQL statement based on the time node timestamp; and package the SQL statement to obtain the change data.

[0112] According to embodiments of this disclosure, any multiple modules among the log monitoring module 710, change data acquisition module 720, flag bit changing module 730, statement conversion module 740, and migration module 750 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the log monitoring module 710, change data acquisition module 720, flag bit changing module 730, statement conversion module 740, and migration module 750 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), programmable logic array (PLA), system-on-a-chip, system-on-a-substrate, system-on-package, application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, and firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the log monitoring module 710, change data acquisition module 720, flag bit changing module 730, statement conversion module 740, and migration module 750 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0113] Figure 8 A block diagram of an electronic device suitable for implementing a bidirectional data migration method according to an embodiment of the present disclosure is shown schematically.

[0114] like Figure 8 As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0115] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0116] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0117] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0118] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0119] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0120] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0121] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0122] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0123] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0125] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0126] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for bidirectional data migration, comprising: Monitor whether the log information in the source database has changed; When the log information changes, the changed data under different data tables in the source database is obtained, wherein the changed data includes a flag field; Change the flag field in the changed data; The changed data is converted into change statements, wherein the change statements are executable at least in the target database; and If the flag field of the change statement is not a default value, a migration operation with the target database is performed based on the change statement; The step of obtaining change data from different data tables in the source database includes: obtaining the data tables involved in the change data; creating topics, segmented channels, and consumer models based on the message middleware, wherein each topic corresponds one-to-one with a data table, each consumer model corresponds one-to-one with a data table, and each topic has a one-to-many relationship with a segmented channel; storing the change data into the topic according to the one-to-one correspondence between the topic and the data table; and retrieving the change data in batches based on the consumer model. The batch includes the first batch. After the migration operation with the target database is performed based on the change statement, the method further includes: checking whether multiple topics under the first batch have been successfully migrated based on the returned migration results of the segmentation channel, wherein the multiple topics include the first topic; if any migration result under the first topic is a failure, the first topic is determined to have failed to migrate; if the first topic is successfully migrated, the first offset of the last migration result of the first topic is recorded, and the first offset is incremented by one before submission; and if the first topic fails to migrate, the second offset of the failed migration result in the segmentation channel is recorded and the second offset is submitted to avoid duplicate migration through the second offset.

2. The method according to claim 1, wherein, The segmented channel includes at least a first channel. Based on the migration results of the returned segmented channels, the process of checking whether multiple topics in the first batch have been successfully migrated includes: For the first channel, iterate through multiple migration results and determine whether any migration result has failed; and If a migration result fails during the traversal of multiple migration results, the current topic containing the migration result is determined to have failed. The traversal of multiple segmentation channels in the current topic is then stopped, and the process proceeds to the next topic.

3. The method according to claim 1, wherein, The change statement is executable in the source database. After converting the changed data into a change statement, the method further includes: backing up the change statement to the source database.

4. The method according to any one of claims 1-3, wherein, The source database includes an Oracle database and a MySQL database, and the target database includes an Oracle database and a MySQL database. When the log information changes, obtaining the changed data under different data tables in the source database includes: For the Oracle database, the materialized view log is parsed to obtain the binary code of the changed object; Query the binary code of the changed object to locate the primary key field, ordinary field, and value of the changed data; The primary key field, the ordinary field, and the value are packaged and encapsulated to obtain the changed data; For the MySQL database, parse the log file to obtain the timestamps of the change timestamps; Based on the timestamp of the aforementioned time point, query the SQL statement; and The SQL statements are packaged to obtain the changed data.

5. A bidirectional data migration device, comprising: The log monitoring module is used to monitor whether the log information in the source database has changed; The change data acquisition module is used to acquire change data under different data tables in the source database when the log information changes, wherein the change data includes a flag field; The flag bit changing module is used to change the flag bit field in the changed data; A statement conversion module is used to convert the changed data into change statements, wherein the change statements are executable at least in the target database; and The migration module is used to perform a migration operation with the target database based on the change statement when the field of the flag bit of the change statement is not a default value; The step of obtaining change data from different data tables in the source database includes: obtaining the data tables involved in the change data; creating topics, segmented channels, and consumer models based on the message middleware, wherein each topic corresponds one-to-one with a data table, each consumer model corresponds one-to-one with a data table, and each topic has a one-to-many relationship with a segmented channel; storing the change data into the topic according to the one-to-one correspondence between the topic and the data table; and retrieving the change data in batches based on the consumer model. The batch includes the first batch. After the migration operation with the target database is performed based on the change statement, the method further includes: checking whether multiple topics under the first batch have been successfully migrated based on the returned migration results of the segmentation channel, wherein the multiple topics include the first topic; if any migration result under the first topic is a failure, the first topic is determined to have failed to migrate; if the first topic is successfully migrated, the first offset of the last migration result of the first topic is recorded, and the first offset is incremented by one before submission; and if the first topic fails to migrate, the second offset of the failed migration result in the segmentation channel is recorded and the second offset is submitted to avoid duplicate migration through the second offset.

6. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 4.

8. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 4.

Images