Cross-platform heterogeneous data synchronization method and device

By comparing incremental, full, and message queue data of the databases on the distributed platform and the host platform, the discrepancies were identified and batch backup synchronization was performed, thus solving the data consistency problem between heterogeneous databases and achieving data consistency synchronization.

CN117633107BActive Publication Date: 2026-11-17CHINA CONSTRUCTION BANK
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202311605585.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-28
Publication Date
2026-11-17
Estimated Expiration
2043-11-28

AI Technical Summary

Technical Problem

Existing technologies struggle to achieve consistent data synchronization between heterogeneous databases, especially in data synchronization between distributed databases and host platform databases, where complex change rules make it difficult to guarantee data consistency.

Method used

By comparing incremental data, full data, and message queue data in the databases of the distributed platform and the host platform, discrepancies are identified, and synchronization is performed through a batch fallback mechanism to ensure data consistency.

Benefits of technology

It achieves data consistency synchronization between heterogeneous databases. By comparing daily incremental, full, and message queue data, it uses a batch fallback mechanism to resynchronize the difference data, ensuring the consistency of data on both sides.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117633107B_ABST
    Figure CN117633107B_ABST
Patent Text Reader

Abstract

This invention discloses a method and apparatus for cross-platform heterogeneous data synchronization. The method includes: after transaction data is updated to the database of a host platform and synchronized to the database of a distributed platform, comparing the daily incremental data of the distributed platform database and the host platform database to obtain a first comparison result; reading and comparing the full data of the distributed platform database and the host platform database to obtain a second comparison result; comparing the message data of the message queues (MQ) of the distributed platform database and the host platform database synchronized daily to obtain a third comparison result; determining the difference data between the distributed platform database and the host platform database based on the first comparison result, the second comparison result, and the third comparison result; and resynchronizing the difference data in the distributed platform database and the host platform database through a batch fallback mechanism. This invention achieves data consistency in heterogeneous data synchronization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for synchronizing heterogeneous data across platforms. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] Traditional data synchronization typically requires consistent table structures and consistent field values ​​or simple rule changes. However, distributed databases have complex change rules, so heterogeneous synchronization comparison can be used for data synchronization. However, heterogeneous comparison is difficult to guarantee data consistency. Summary of the Invention

[0004] This invention provides a cross-platform heterogeneous data synchronization method to achieve data consistency in heterogeneous data synchronization. The method includes:

[0005] After the transaction data is updated to the database on the host platform and synchronized to the database on the distributed platform, the daily incremental data of the database on the distributed platform and the database on the host platform are compared to obtain the first comparison result.

[0006] Read and compare the full data of the database on the distributed platform and the database on the host platform to obtain the second comparison result;

[0007] The message data of the message queue MQ of the distributed platform's database and the host platform's database, which are synchronized daily, are compared to obtain the third comparison result;

[0008] Based on the first comparison result, the second comparison result, and the third comparison result, the differences between the database of the distributed platform and the database of the host platform are determined;

[0009] The batch backup mechanism is used to resynchronize the discrepancies between the databases on the distributed platform and the host platform.

[0010] This invention also provides a cross-platform heterogeneous data synchronization device to achieve data consistency in heterogeneous data synchronization. The device includes:

[0011] The comparison module is used to compare the daily incremental data of the distributed platform database and the host platform database after the transaction data is updated to the host platform database and synchronized to the distributed platform database to obtain the first comparison result; read and compare the full data of the distributed platform database and the host platform database to obtain the second comparison result; and compare the message data of the message queue MQ of the distributed platform database and the host platform database synchronized daily to obtain the third comparison result.

[0012] The synchronization module is used to determine the differences between the database of the distributed platform and the database of the host platform based on the first comparison result, the second comparison result, and the third comparison result; and to resynchronize the differences between the database of the distributed platform and the database of the host platform through a batch fallback mechanism.

[0013] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described cross-platform heterogeneous data synchronization method.

[0014] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described cross-platform heterogeneous data synchronization method.

[0015] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described cross-platform heterogeneous data synchronization method.

[0016] In this embodiment of the invention, after transaction data is updated to the database of the host platform and synchronized to the database of the distributed platform, the daily incremental data of the databases of the distributed platform and the host platform are compared to obtain a first comparison result; the full data of the databases of the distributed platform and the host platform are read and compared to obtain a second comparison result; the message data of the message queues (MQ) of the databases of the distributed platform and the host platform synchronized daily are compared to obtain a third comparison result; the difference data between the databases of the distributed platform and the host platform are determined based on the first, second, and third comparison results; and the difference data is resynchronized in the databases of the distributed platform and the host platform through a batch fallback mechanism. Compared with the prior art, by comparing the daily incremental data, the full data, and the daily synchronized MQ message data, and by resynchronizing the difference data through batch fallback, data consistency in heterogeneous data synchronization is achieved. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0018] Figure 1A flowchart illustrating the cross-platform heterogeneous data synchronization method provided by this invention;

[0019] Figure 2 A flowchart illustrating the cross-platform heterogeneous data synchronization method provided by this invention;

[0020] Figure 3 A flowchart illustrating the cross-platform heterogeneous data synchronization method provided by this invention;

[0021] Figure 4 A flowchart illustrating the cross-platform heterogeneous data synchronization method provided by this invention;

[0022] Figure 5 This is a schematic diagram of the cross-platform heterogeneous data synchronization device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0024] Figure 1 This is a flowchart illustrating the cross-platform heterogeneous data synchronization method provided in this embodiment of the invention, as shown below. Figure 1 As shown, the method includes:

[0025] Step 101: After the transaction data is updated to the database of the host platform and synchronized to the database of the distributed platform, the daily incremental data of the database of the distributed platform and the database of the host platform are compared to obtain the first comparison result.

[0026] Step 102: Read and compare the full data of the database on the distributed platform and the database on the host platform to obtain the second comparison result.

[0027] Step 103: Compare the message data of the message queue MQ of the distributed platform's database and the host platform's database, which are synchronized daily, to obtain the third comparison result.

[0028] Step 104: Determine the differences between the database of the distributed platform and the database of the host platform based on the first comparison result, the second comparison result, and the third comparison result.

[0029] Step 105: Use a batch fallback mechanism to resynchronize the discrepancies between the database on the distributed platform and the database on the host platform.

[0030] The above solution ensures data consistency between the two sides by comparing daily incremental data, full data of key tables, and daily synchronized MQ message data, and by using batch backup to resynchronize differing data. The key point is achieving data consistency in heterogeneous data synchronization by comparing incremental and full data of key tables and daily synchronized MQ messages, and by using batch backup to resynchronize differing data.

[0031] In step 101 of this embodiment of the invention, the daily incremental data of the database on the distributed platform and the database on the host platform are compared to obtain a first comparison result. The steps are as follows: Figure 2 As shown, the details are as follows:

[0032] Step 201: Read the synchronization record table page by page according to the transaction date.

[0033] It should be noted that the synchronization record table is obtained after the transaction data is synchronized from the database on the host platform to the database on the distributed platform.

[0034] Step 202: Read field information from the databases of the distributed platform and the host platform respectively based on the key value in the synchronization record table. Step 203: Compare the field information to obtain the first comparison result.

[0035] During the comparison, each table has a corresponding comparison configuration because the field settings are different. The field information is read from the database of the distributed platform and the database of the host platform respectively, and then compared according to the specific table configuration.

[0036] In this embodiment of the invention, the synchronous record table is read in pages according to the business day. The records read in a loop are read from both the distributed MySQL database and the database of the host mainframe platform according to the key value. The main logic of data comparison is called to compare the data according to the fields.

[0037] In one possible implementation, the first comparison result includes the number of consistent and inconsistent records, as well as details of the specific inconsistent records.

[0038] In step 102 of this embodiment of the invention, the full data of the database of the distributed platform and the database of the host platform are read and compared to obtain a second comparison result. The steps are as follows: Figure 3 As shown, the details are as follows:

[0039] Step 301: Read the pre-configured priority table.

[0040] Step 302: Calculate the key values ​​corresponding to the database of the host platform based on the pre-configured key tables.

[0041] Read the key tables that have successfully completed distributed synchronization, and calculate the corresponding key values ​​in the host platform's DB2 database based on these key tables.

[0042] Step 303: Read the database of the host platform according to the key value corresponding to the database of the host platform.

[0043] In one possible implementation, the record corresponding to the host platform is read via DRDA based on the key value corresponding to the host platform's database.

[0044] It should be noted that the Distributed Relational Database Architecture (DRD) is an IBM standard for accessing database information across IBM platforms and conforming to the SQL standard.

[0045] Step 304: Compare the database of the distributed platform with the database of the host platform that has been read to obtain the second comparison result.

[0046] In this embodiment of the invention, the key table records configured are read in pages, the corresponding KEY value of host DB2 is calculated based on the read records, the records of DB2 are read through drda, the main data comparison logic is called, and the comparison is performed according to the fields.

[0047] In step 103 of this embodiment of the invention, the message data of the message queue MQ of the distributed platform's database and the host platform's database, which are synchronized daily, are compared to obtain a third comparison result. The process is as follows: Figure 4 As shown, the details are as follows:

[0048] Step 401: The first MQ data file sent by the host platform to the data synchronization cluster via the MQ message mechanism is sent to the distributed file system.

[0049] Step 402: Send the second MQ data file received by the distributed platform to the distributed file system.

[0050] Step 403: The first MQ data file and the second MQ data file in the distributed file system are compared using the Spark platform to obtain the third comparison result.

[0051] In this embodiment of the invention, the host intercepts data through the "unified database access interface", obtains the data to be added and modified, sends it to the data synchronization cluster through the MQ message mechanism, and synchronously updates the key value of the sent information to the database. The host downloads the MQ data sent on the same day in batch time periods and sends it to the distributed HDFS. The heterogeneous synchronization compares the MQ data received on the same day and puts it into the distributed HDFS. Spark is used to compare the records of the two files.

[0052] If a distributed data loss is detected during the comparison, the distributed system will return the MQ key-value file to the host. The host will then add a new batch and, if the MQ key-value file sent from the distributed system to the host exists, execute a resend MQ job to ensure data consistency between the two sides.

[0053] This invention also provides a cross-platform heterogeneous data synchronization device, as described in the following embodiments. This device is as follows... Figure 5 As shown, the device includes:

[0054] The comparison module 501 is used to compare the daily incremental data of the distributed platform database and the host platform database after the transaction data is updated to the host platform database and synchronized to the distributed platform database to obtain the first comparison result; read and compare the full data of the distributed platform database and the host platform database to obtain the second comparison result; and compare the message data of the message queue MQ of the distributed platform database and the host platform database synchronized daily to obtain the third comparison result.

[0055] The synchronization module 502 is used to determine the difference data between the database of the distributed platform and the database of the host platform based on the first comparison result, the second comparison result, and the third comparison result; and to resynchronize the difference data between the database of the distributed platform and the database of the host platform through a batch fallback mechanism.

[0056] In this embodiment of the invention, the comparison module 501 is specifically used for:

[0057] The synchronization record table is read page by page based on the transaction date; the synchronization record table is obtained after the transaction data is synchronized from the database on the host platform to the database on the distributed platform.

[0058] Based on the key value in the synchronization record table, field information is read from the database of the distributed platform and the database of the host platform respectively;

[0059] The first comparison result is obtained by comparing the field information.

[0060] In this embodiment of the invention, the comparison module 501 is specifically used for:

[0061] Read the pre-configured priority table;

[0062] Calculate the corresponding key values ​​in the host platform's database based on the pre-configured key tables;

[0063] Read the database of the host platform based on the key value corresponding to the database of the host platform;

[0064] The database of the distributed platform is compared with the database of the host platform, and a second comparison result is obtained.

[0065] In this embodiment of the invention, the comparison module 501 is specifically used for:

[0066] The host platform sends the first MQ data file, which is sent to the data synchronization cluster via the MQ message mechanism, to the distributed file system.

[0067] Send the second MQ data file received by the distributed platform to the distributed file system;

[0068] The Spark platform is used to compare the first and second MQ data files in the distributed file system to obtain the third comparison result.

[0069] Since the principle behind this device for solving the problem is similar to that of the cross-platform heterogeneous data synchronization method, the implementation of this device can be found in the implementation of the cross-platform heterogeneous data synchronization method, and the repetitive parts will not be repeated.

[0070] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described cross-platform heterogeneous data synchronization method.

[0071] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described cross-platform heterogeneous data synchronization method.

[0072] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described cross-platform heterogeneous data synchronization method.

[0073] In this embodiment of the invention, after transaction data is updated to the database of the host platform and synchronized to the database of the distributed platform, the daily incremental data of the databases of the distributed platform and the host platform are compared to obtain a first comparison result; the full data of the databases of the distributed platform and the host platform are read and compared to obtain a second comparison result; the message data of the message queues (MQ) of the databases of the distributed platform and the host platform synchronized daily are compared to obtain a third comparison result; the difference data between the databases of the distributed platform and the host platform are determined based on the first, second, and third comparison results; and the difference data is resynchronized in the databases of the distributed platform and the host platform through a batch fallback mechanism. Compared with the prior art, by comparing the daily incremental data, the full data, and the daily synchronized MQ message data, and by resynchronizing the difference data through batch fallback, data consistency in heterogeneous data synchronization is achieved.

[0074] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0078] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cross-platform heterogeneous data synchronization method, characterized in that, include: After the transaction data is updated to the host platform's database and synchronized to the distributed platform's database, the daily incremental data of the distributed platform's database and the host platform's database are compared to obtain the first comparison result. This includes: reading the synchronization record table page by page according to the transaction date; the synchronization record table is obtained after the transaction data is synchronized from the host platform's database to the distributed platform's database; reading field information from the distributed platform's database and the host platform's database respectively according to the key value in the synchronization record table; and comparing the field information to obtain the first comparison result. The process involves reading and comparing the full data of the distributed platform's database and the host platform's database to obtain a second comparison result. This includes: reading pre-configured key tables; calculating the corresponding key values ​​of the host platform's database based on the pre-configured key tables; reading the host platform's database based on the corresponding key values; and comparing the distributed platform's database with the read host platform's database to obtain the second comparison result. The message data of the message queue MQ of the distributed platform's database and the host platform's database, which are synchronized daily, are compared to obtain the third comparison result; Based on the first comparison result, the second comparison result, and the third comparison result, the differences between the database of the distributed platform and the database of the host platform are determined; By using a batch backup mechanism, the discrepancies are resynchronized between the databases on the distributed platform and the host platform to achieve data consistency in heterogeneous data synchronization.

2. The cross-platform heterogeneous data synchronization method as described in claim 1, characterized in that, By comparing the message data of the message queues (MQ) of the distributed platform's database and the host platform's database, which are synchronized daily, a third comparison result is obtained, including: The host platform sends the first MQ data file, which is sent to the data synchronization cluster via the MQ message mechanism, to the distributed file system. Send the second MQ data file received by the distributed platform to the distributed file system; The Spark platform is used to compare the first and second MQ data files in the distributed file system to obtain the third comparison result.

3. A cross-platform heterogeneous data synchronization device, characterized in that, include: The comparison module is used to compare the daily incremental data of the distributed platform's database and the host platform's database after transaction data is updated to the host platform's database and synchronized to the distributed platform's database to obtain a first comparison result. This includes: reading the synchronization record table paginated according to the transaction date; obtaining the synchronization record table after the transaction data is synchronized from the host platform's database to the distributed platform's database; reading field information from both the distributed platform's database and the host platform's database based on the key values ​​in the synchronization record table; comparing the field information to obtain the first comparison result; reading and comparing the full data of the distributed platform's database and the host platform's database to obtain a second comparison result, which includes: reading a pre-configured key table; calculating the corresponding key value in the host platform's database based on the pre-configured key table; reading the host platform's database based on the corresponding key value; comparing the distributed platform's database with the read host platform's database to obtain the second comparison result; and comparing the message data of the message queues (MQ) of the daily synchronized distributed platform's database and the host platform's database to obtain a third comparison result. The synchronization module is used to determine the differences between the database of the distributed platform and the database of the host platform based on the first comparison result, the second comparison result, and the third comparison result; and to resynchronize the differences between the database of the distributed platform and the database of the host platform through a batch fallback mechanism to achieve data consistency of heterogeneous data synchronization.

4. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 2.

6. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 2.

Citation Information

Patent Citations

  • Cross-database platform synchronous data checking method, device and system

    CN110647534A