Data synchronous reconciliation method, system and device for heterogeneous database and storage medium

Abstract

The embodiment of the invention discloses a data synchronous reconciliation method, system and device for a heterogeneous database and a storage medium. According to the technical scheme provided by the embodiment of the invention, the data change record of the first database is acquired, the data change record is written into the corresponding data group according to the same data identifier, and the second database is queried based on the data identifier of the current data group under the condition that the data reconciliation of the current data group is triggered. Determining target data matched with the data identifier of the current data group, and comparing the version number of the current data group based on the version number of the matched target data; and under the condition that the version number of the current data packet is the same as the version number of the target data, determining that the data synchronization reconciliation of the current data packet is successful. By adopting the technical means, accurate data synchronous account checking between heterogeneous databases can be realized, abnormal business operation caused by data asynchronization is avoided, and the reliability and the stability of business processing are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method, system, device and storage medium for data synchronization and reconciliation of heterogeneous databases. Background Art

[0002] At present, in a business system, in order to meet the performance requirements of different business scenarios, different database systems are often used to store and query data. Different databases have different characteristics and advantages. When multiple heterogeneous databases need to be applied, data synchronization between heterogeneous databases is required. For example, in the work order grabbing system of the labeling platform, the labeler enters complex screening conditions in the front-end interface, and the back-end system uses the powerful search capabilities of Elasticsearch (open source distributed search and data storage engine) to quickly filter qualified work orders, and then judge and update the dispatch status of this batch of work orders in MySQL (relational database management system), and then synchronize this data change to Elasticsearch to ensure the synchronization of data status.

[0003] However, during the data synchronization process between heterogeneous databases, data synchronization failures may easily occur due to network or machine failures, and the backend service cannot detect the data synchronization failure in time, which leads to data inconsistency between heterogeneous databases, affecting business operations and causing poor business execution results. Summary of the invention

[0004] The embodiments of the present application provide a data synchronization reconciliation method, system, device and storage medium for heterogeneous databases, which can accurately reconcile data synchronization between heterogeneous databases and solve the technical problem of inconsistent data in heterogeneous databases.

[0005] In a first aspect, an embodiment of the present application provides a data synchronization and reconciliation method for a heterogeneous database, comprising:

[0006] Obtaining a data change record of the first database, the data change record including a data identifier and a version number, the first database generates a data change record when synchronizing target data to the second database, the target data including the corresponding data identifier and version number;

[0007] Writing the data change record into the corresponding data group according to the same data identifier, and in the case of triggering data reconciliation of the current data group, querying the second database based on the data identifier of the current data group, determining the target data matching the data identifier of the current data group, and comparing the version number of the current data group based on the version number of the matched target data;

[0008] When the current data group has a version number that is the same as the version number of the target data, it is determined that the data synchronization reconciliation of the current data group is successful.

[0009] In a second aspect, an embodiment of the present application provides a data synchronization and reconciliation system for a heterogeneous database, including:

[0010] an acquisition module, configured to acquire a data change record of the first database, the data change record including a data identifier and a version number, the first database generates a data change record when synchronizing target data to the second database, the target data including the corresponding data identifier and version number;

[0011] a query module configured to write the data change record into the corresponding data group according to the same data identifier, and in the case of triggering data reconciliation of the current data group, query the second database based on the data identifier of the current data group, determine the target data matching the data identifier of the current data group, and compare the version number of the current data group based on the version number of the matched target data;

[0012] The reconciliation module is configured to determine that the data synchronization reconciliation of the current data group is successful when the current data group has a version number that is the same as the version number of the target data.

[0013] In a third aspect, an embodiment of the present application provides a data synchronization and reconciliation device for a heterogeneous database, including:

[0014] memory and one or more processors;

[0015] The memory is configured to store one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the data synchronization and reconciliation method for heterogeneous databases as described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions, when executed by a computer processor, are configured to execute the data synchronization reconciliation method for heterogeneous databases as described in the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed on a computer or a processor, the computer or the processor executes the data synchronization and reconciliation method of heterogeneous databases as described in the first aspect.

[0019] The embodiment of the present application obtains the data change record of the first database, the data change record includes a data identifier and a version number, the first database generates a data change record when synchronizing the target data to the second database, the target data includes the corresponding data identifier and version number; the data change record is written into the corresponding data group according to the same data identifier, and when the data reconciliation of the current data group is triggered, the second database is queried based on the data identifier of the current data group, the target data matching the data identifier of the current data group is determined, and the version number of the current data group is compared based on the version number of the matched target data; when the current data group has the same version number as the version number of the target data, it is determined that the data synchronization reconciliation of the current data group is successful. By adopting the above-mentioned technical means, the target data is queried by the data identifier, the version number of the data group is compared based on the version number of the target data, and the data synchronization reconciliation is performed according to the comparison result, so as to realize the accurate data synchronization reconciliation between heterogeneous databases, avoid abnormal business operation caused by data asynchrony, and improve the reliability and stability of business processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of a data synchronization and reconciliation method for a heterogeneous database provided in an embodiment of the present application;

[0021] Figure 2 Schematic diagram of data query and synchronization of heterogeneous databases in an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of data synchronization and reconciliation of heterogeneous databases in an embodiment of the present application;

[0023] Figure 4 It is a structural diagram of a data synchronization and reconciliation system for heterogeneous databases provided in an embodiment of the present application;

[0024] Figure 5 It is a structural diagram of a data synchronization and reconciliation device for a heterogeneous database provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0026] The data synchronization and reconciliation method for heterogeneous databases provided in the present application aims to query target data through data identifiers, compare the version numbers of data groups based on the version numbers of the target data, and perform data synchronization and reconciliation according to the comparison results, so as to achieve accurate data synchronization and reconciliation between heterogeneous databases.

[0027] In business systems, different database systems are often used to meet the performance requirements of different business scenarios. Different databases have different characteristics and advantages. For example, in the work order grabbing system of the labeling platform, the labeler enters complex screening conditions in the front-end interface. The back-end system uses the powerful search capabilities of Elasticsearch (distributed search and analysis engine) to quickly filter qualified work orders, and then judges and updates the dispatch status of this batch of work orders in MySQL (relational database management system), and then synchronizes the updated data to Elasticsearch. This process involves the synchronization of data between different databases. If Elasticsearch fails to synchronize data with MySQL due to network or machine failures, the back-end service may not be able to perceive the synchronization failure. For example, if data is only synchronized to the primary shard of Elasticsearch, and the primary shard fails when the replica does not synchronize data, the data will be lost. As time goes by, the data inconsistency between MySQL and Elasticsearch will increase, resulting in the work orders selected by Elasticsearch not meeting expectations, and the problem of wrong order grabbing, which in turn affects the availability of the labeling business.

[0028] To this end, in the relevant data synchronization and reconciliation scheme, a data comparison method based on the local backend database and the cloud database is adopted, and the data written within the preset time period is obtained from the local backend database and the cloud database respectively for data comparison at both ends. Since the direct connection database batch obtains the data set to be compared within the preset time period, when the magnitude of the data to be compared within the preset time period is large, reading a large amount of data from the database in a short time will cause fluctuations in the database CPU, memory, IO and other indicators, affecting the stability of the database, and there is a large risk of causing the business system to be unavailable; in addition, since the data is constantly changing, when comparing the state of the data at a certain point in time, there is a time difference between reading the data from the local database and reading the data from the cloud database. If the data state changes within this time difference, it will cause reconciliation failure, but this situation is not due to inconsistency in the data itself, but because the reconciliation data set expires and causes failure. Frequent triggering of reconciliation failures has the problem of low confidence in the reconciliation mechanism. Obviously, the final state of the data can be determined by replaying the data change log. It is not necessary to reconcile the final state of the data. The purpose of data reconciliation can be achieved by comparing the data change log.

[0029] Based on this, a data synchronization and reconciliation method for heterogeneous databases of an embodiment of the present application is provided. Through a universal data synchronization and real-time reconciliation and repair solution between heterogeneous databases, data synchronization of heterogeneous database systems is achieved and the final consistency of synchronized data is guaranteed. In the reconciliation process, it has both strong robustness and closed-loop automatic data repair capabilities, solving the technical problem of inconsistent data in heterogeneous databases.

[0030] Example:

[0031] Figure 1 A flow chart of a data synchronization and reconciliation method for a heterogeneous database provided in an embodiment of the present application is given. The data synchronization and reconciliation method for a heterogeneous database provided in this embodiment can be executed by a data synchronization and reconciliation device for a heterogeneous database. The data synchronization and reconciliation device for a heterogeneous database can be implemented by software and / or hardware. The data synchronization and reconciliation device for a heterogeneous database can be composed of two or more physical entities, or can be composed of one physical entity. Generally speaking, the data synchronization and reconciliation device for a heterogeneous database can be a computing device such as a data synchronization server.

[0032] The following description is made by taking the data synchronization and reconciliation device of the heterogeneous database as the main body of the data synchronization and reconciliation method for executing the heterogeneous database as an example. Figure 1 The data synchronization and reconciliation method of the heterogeneous database specifically includes:

[0033] S110, obtaining a data change record of the first database, the data change record including a data identifier and a version number, the first database generates a data change record when synchronizing target data to the second database, the target data including the corresponding data identifier and version number.

[0034] When performing data synchronization reconciliation, this application performs data synchronization reconciliation based on the version number of the data change record, and accurately determines whether the data is synchronized between heterogeneous databases by comparing the version numbers.

[0035] For example, take the data synchronization and reconciliation between heterogeneous databases Elasticsearch and MySQL as an example. Figure 2 In the order grabbing system, the labeler enters the work order screening conditions in the front-end interface, and uses the data search capabilities of the heterogeneous database Elasticsearch and the relational data storage characteristics of MySQL to query, store and synchronize the work order data. Define MySQL as the first database and Elasticsearch as the second database. The back-end system will use Elasticsearch (the second database) to quickly screen the work orders that meet the conditions, and then judge and update the dispatch status of this batch of work orders in MySQL (the first database). The first database returns the work order to the user according to the screening conditions. At the same time, MySQL (the first database) synchronizes the updated work order status to Elasticsearch (the second database).

[0036] During the synchronization of the above work order status, it is necessary to reconcile the data synchronization between MySQL and Elasticsearch to ensure that the updated work order status is successfully synchronized to Elasticsearch and to ensure the consistency of data storage. Therefore, when using Elasticsearch in the backend to quickly retrieve qualified work orders in MySQL according to the screening conditions, MySQL will update the order grabbing status of the work order in MySQL according to the screening results, and generate a change record containing the data identifier (such as the work order ID) and version number. Then write the change record to the Kafka message queue to ensure reliable data transmission and storage. Kafka provides a high availability mechanism to ensure that data is not lost due to a single point of failure. Then, the message in Kafka is consumed through the data synchronization module, and the change record is first written to the data synchronization reconciliation device, and then synchronized to Elasticsearch to complete the synchronization of the work order status.

[0037] Optionally, for data change records, each data change record may include a unique ID of the data (ie, data identifier), a version number, a current data state, and a conversion rule, wherein the conversion rule identifies change information such as addition, deletion, and modification of the data.

[0038] S120. Write the data change record into the corresponding data group according to the same data identifier, and when triggering data reconciliation of the current data group, query the second database based on the data identifier of the current data group to determine the target data that matches the data identifier of the current data group, and compare the version number of the current data group based on the version number of the matched target data.

[0039] For data synchronization and reconciliation equipment, the received change records will be grouped according to their data identifiers, and the same ID will enter the same group. Then, according to the actual reconciliation needs, the data reconciliation operation of the corresponding data group is adaptively triggered. For example, every preset time period T1 (indicating the time interval for triggering each round of reconciliation), the reconciliation operation of the current data group is triggered. Optionally, the data synchronization and reconciliation device can also trigger the reconciliation operation based on information such as the number of data change records accumulated in the data group.

[0040] Reference Figure 3 In steps a1-a4, after the first database writes the data change record and synchronizes the data to the second database according to the data change record, if data synchronization reconciliation is triggered, the ID (data identifier) ​​of the change record of the current data group is obtained, the state of the current data is obtained from Elasticsearch according to the ID, and the record corresponding to the ID can be found in Elasticsearch, and the record matching the data identifier of the current data group is defined as the target data. The version number of the target data in Elasticsearch is extracted and compared with the version number of the data change record of the current data group, so as to perform data synchronization reconciliation.

[0041] It can be understood that if the version number of the target data is higher than or equal to the version number of the current data group, it means that the data in the target database is the latest, or at least as new as the current data group, that is, the data synchronization reconciliation is successful. If the version number of the target data is lower than the version number of the current data group, it means that the current data group contains updated data, and this part of the data has not been updated to Elasticsearch. At this time, it is necessary to further determine whether the data synchronization is abnormal.

[0042] S130: When the current data group has a version number that is the same as the version number of the target data, determine that the data synchronization reconciliation of the current data group is successful.

[0043] Furthermore, the version number of the current data group is compared based on the version number of the target data, and by traversing the change records in the current ID group, an attempt is made to match the change records with the same version number as the Elasticsearch data. If in the change records of the current data group, there is data with the same version number as the target data in Elasticsearch, then the data synchronization and reconciliation between the two is successful. Optionally, in order to improve the accuracy of data synchronization and reconciliation, the data status of the two parts of data can be further compared. If the status of the data with the same version number is exactly the same, the reconciliation is determined to be successful. This application does not impose fixed restrictions on the specific comparison content of data synchronization and reconciliation, and will not be elaborated here.

[0044] It should be noted that, when the current data group has the same version number as the target data, the version number of the target data may be smaller than the maximum version number of the current data group. This situation may be caused by data synchronization delay, that is, the change record of the first database is updated to the data group, but not updated to Elasticsearch. At this time, the data reconciliation is still determined to be successful.

[0045] In addition, this application is applicable to data consistency verification between heterogeneous databases, including database migration and data synchronization, not limited to MySQL and Elasticsearch, but also applicable to MySQL and TiDB. On the other hand, the message queue of this application is not limited to Kafka, and this application does not impose fixed restrictions on the application types of heterogeneous databases and message queues, which will not be elaborated here.

[0046] Optionally, after determining that the data synchronization reconciliation of the current data group is successful, the method further includes:

[0047] Determine the maximum version number of the current data group, and delete the current data group when the version number of the target data is equal to the maximum version number.

[0048] The maximum version number of the current data group is determined, and when the version number of the target data is smaller than the maximum version number, the data change record whose version number is smaller than the version number of the target data is deleted from the current data group.

[0049] If the version number of the target data that matches the current data group is equal to the maximum version number in the change record of the current data group, the current ID group needs to be deleted. It can be understood that when the version number of the target data is equal to the maximum version number in the change record of the current data group, it means that all change records of the current data group have been synchronized to Elasticsearch and are no longer needed for subsequent data reconciliation. In order to save data storage space, the current ID group is deleted at this time to optimize the storage of data change records.

[0050] On the other hand, if the version number of the target data that matches the current data group is smaller than the maximum version number in the change record in the current ID group, the change records in the current ID group that are smaller than the matching version number are deleted, and the remaining change records are used for the next data synchronization reconciliation. Similarly, the change records that are smaller than the matching version number have been synchronized to Elasticsearch and are no longer needed for subsequent data reconciliation. In order to save data storage space, this part of the data is deleted at this time to optimize the storage of data change records.

[0051] Optionally, after comparing the version number of the current data group based on the version number of the matched target data, the method further includes:

[0052] When the current data group does not have a version number identical to that of the target data, the corresponding target data is resynchronized from the first database to the second database.

[0053] If there is no change record with the same version number as the Elasticsearch data in the change record of the current ID group, it means that the change record of the current data group has not been synchronized to Elasticsearch. Figure 3 , by notifying the first database mysql, re-update the work order status to Elasticsearch (the second database). Then, during the next reconciliation, if the current ID group has not changed, the change record of the current ID group contains the change record of the maximum version number of the retry, and the version number of the Elasticsearch data is also the retry version number, then the version number of the matched Elasticsearch target data will be equal to the maximum version number in the change record of the current data group, which corresponds to the above-mentioned situation where the version numbers are the same after the data reconciliation is successful. If the current ID group has changed, the version number of the matched Elasticsearch target data may be less than or equal to the maximum version number in the change record of the current data group, or there may be no change record with the same version number as the Elasticsearch data. But in either case, when the data stops changing within a reconciliation cycle, it will always converge to the situation where the data synchronization reconciliation is successful, and the version number of the matched Elasticsearch target data will be equal to the maximum version number in the change record of the current data group.

[0054] Optionally, after querying the second database based on the data identifier of the current data group, the method further includes:

[0055] When it is determined that the data identifier of the current data group does not have matching target data, and the data change rule of the data change record corresponding to the maximum version number of the current data group is deletion, it is determined that the data synchronization reconciliation of the current data group is successful, and the current data group is deleted.

[0056] By querying Elasticsearch, if the record corresponding to the current ID cannot be found in Elasticsearch, and the conversion rule of the change record corresponding to the maximum version number of the current data group is deletion, then the reconciliation is successful and the current ID group is deleted. It can be understood that when the conversion rule of the data change record is deletion, Elasticsearch will delete the corresponding ID data according to the data change record, so the record corresponding to the current ID cannot be found in Elasticsearch, which means that the data synchronization reconciliation is successful.

[0057] Optionally, after querying the second database based on the data identifier of the current data group, the method further includes:

[0058] When it is determined that there is no matching target data for the data identifier of the current data group, and the data change rule of the data change record corresponding to the maximum version number of the current data group is addition or modification, it is determined that the data synchronization reconciliation of the current data group is successful.

[0059] By querying Elasticsearch, if the record corresponding to the current ID cannot be found in Elasticsearch, and the conversion rule of the change record corresponding to the maximum version number of the current data group is add or modify, then the reconciliation is successful and the current change record is retained for the next reconciliation. Since the data conversion rule is add or modify, that is, Elasticsearch does not have data corresponding to the ID before, the data reconciliation is successful.

[0060] In addition, when the data reconciliation of the current data group is triggered, it also includes:

[0061] Monitor the maximum version number of the current data group in real time. When it is detected that the maximum version number has not changed within a first set time period, re-synchronize the corresponding target data from the first database to the second database; when it is detected that the maximum version number has not changed within a second set time period, output a business notification to prompt manual repair of the target data corresponding to the first database, and the second set time period is greater than the first set time period.

[0062] By monitoring the maximum version number of the current ID group, if the maximum version number has not changed within T2 (the first set period, indicating the time interval for monitoring whether the maximum version number has changed), the status of the current data is obtained from MySQL according to the ID and resynchronized to Elasticsearch; if there is no change within T3 (the second set period, indicating the maximum time interval for abnormal reconciliation result data), it is necessary to notify the business in time so that the data can be manually repaired.

[0063] Based on the above-mentioned data synchronization and reconciliation mechanism, as long as the data has a unique ID and version number, the reconciliation mechanism of the present application can be used to realize data reconciliation; and, considering that the change record is first written to the data synchronization and reconciliation device, and then synchronized to El asticsearch. When reconciling, the data is first read from El asticsearch, and then compared with the change record in the current ID group. Therefore, under normal circumstances, the version number of the change record in the ID group will be greater than or equal to the version number of the El asticsearch data. In most cases, the version number in the reconciliation mechanism will be equal or the target data version number will be less than the maximum version number of the current data group. Unless the message fails to be written to the data synchronization and reconciliation device, it is necessary to query MySQL to resynchronize the data to El asticsearch. Therefore, the problem of frequently querying the source data during the reconciliation process, which affects the stability of the source database, can be solved.

[0064] In addition, according to the state machine theory, as long as the version number of Elasticsearch can catch up with the change record of the current ID group, the data between the databases will reach final consistency. It can avoid the problem of time difference in the solution of pulling data from the source database and the target database, which leads to reconciliation failure. At the same time, combined with the abnormal data monitoring and repair mechanism, it has a certain fault tolerance for synchronization failure or data loss, and improves the robustness of the reconciliation mechanism. Since the reconciliation source data adopts the change log and the reconciliation target data adopts the method of reading the target database, the defect of one-by-one comparison under the double-log reconciliation mode is avoided, especially in the scenario of less reading and more writing, such as the current ID has changed 1000 times within the preset time, and the double-log reconciliation method needs to be compared 1000 times, while this application only needs to be compared once, which can significantly improve the efficiency of reconciliation.

[0065] In the above, by obtaining the data change record of the first database, the data change record includes a data identifier and a version number. The first database generates a data change record when synchronizing the target data to the second database, and the target data includes the corresponding data identifier and version number; the data change record is written into the corresponding data group according to the same data identifier, and when the data reconciliation of the current data group is triggered, the second database is queried based on the data identifier of the current data group to determine the target data that matches the data identifier of the current data group, and the version number of the current data group is compared based on the version number of the matched target data; when the current data group has the same version number as the version number of the target data, it is determined that the data synchronization reconciliation of the current data group is successful. By adopting the above technical means, the target data is queried by the data identifier, the version number of the data group is compared based on the version number of the target data, and the data synchronization reconciliation is performed according to the comparison result, so as to realize the accurate data synchronization reconciliation between heterogeneous databases, avoid abnormal business operation caused by data asynchrony, and improve the reliability and stability of business processing.

[0066] Based on the above embodiments, Figure 4 A schematic diagram of the structure of a data synchronization and reconciliation system for heterogeneous databases provided in this application. Figure 4 The data synchronization and reconciliation system for heterogeneous databases provided in this embodiment specifically includes: an acquisition module 21, a query module 22 and a reconciliation module 23.

[0067] The acquisition module is configured to acquire a data change record of the first database, the data change record includes a data identifier and a version number, the first database generates a data change record when synchronizing target data to the second database, and the target data includes the corresponding data identifier and version number;

[0068] a query module configured to write the data change record into the corresponding data group according to the same data identifier, and in the case of triggering data reconciliation of the current data group, query the second database based on the data identifier of the current data group, determine the target data matching the data identifier of the current data group, and compare the version number of the current data group based on the version number of the matched target data;

[0069] The reconciliation module is configured to determine that the data synchronization reconciliation of the current data group is successful when the current data group has a version number that is the same as the version number of the target data.

[0070] Specifically, after comparing the version number of the current data group based on the version number of the matched target data, the method further includes:

[0071] When the current data group does not have a version number identical to that of the target data, the corresponding target data is resynchronized from the first database to the second database.

[0072] Specifically, after determining that the data synchronization reconciliation of the current data group is successful, the method further includes:

[0073] Determine the maximum version number of the current data group, and delete the current data group when the version number of the target data is equal to the maximum version number.

[0074] Specifically, after determining that the data synchronization reconciliation of the current data group is successful, the method further includes:

[0075] The maximum version number of the current data group is determined, and when the version number of the target data is smaller than the maximum version number, the data change record whose version number is smaller than the version number of the target data is deleted from the current data group.

[0076] Specifically, after querying the second database based on the data identifier of the current data group, the method further includes:

[0077] When it is determined that the data identifier of the current data group does not have matching target data, and the data change rule of the data change record corresponding to the maximum version number of the current data group is deletion, it is determined that the data synchronization reconciliation of the current data group is successful, and the current data group is deleted.

[0078] Specifically, after querying the second database based on the data identifier of the current data group, the method further includes:

[0079] When it is determined that there is no matching target data for the data identifier of the current data group, and the data change rule of the data change record corresponding to the maximum version number of the current data group is addition or modification, it is determined that the data synchronization reconciliation of the current data group is successful.

[0080] Specifically, in the case of triggering data reconciliation of the current data group, it also includes:

[0081] The maximum version number of the current data group is monitored in real time, and when it is detected that the maximum version number has not changed within a first set period of time, the corresponding target data is resynchronized from the first database to the second database.

[0082] After real-time monitoring of the maximum version number of the current data group, it also includes:

[0083] When it is detected that the maximum version number has not changed within a second set period, a service notification is output to prompt manual repair of the target data corresponding to the first database, and the second set period is greater than the first set period.

[0084] In the above, by obtaining the data change record of the first database, the data change record includes a data identifier and a version number. The first database generates a data change record when synchronizing the target data to the second database, and the target data includes the corresponding data identifier and version number; the data change record is written into the corresponding data group according to the same data identifier, and when the data reconciliation of the current data group is triggered, the second database is queried based on the data identifier of the current data group to determine the target data that matches the data identifier of the current data group, and the version number of the current data group is compared based on the version number of the matched target data; when the current data group has the same version number as the version number of the target data, it is determined that the data synchronization reconciliation of the current data group is successful. By adopting the above technical means, the target data is queried by the data identifier, the version number of the data group is compared based on the version number of the target data, and the data synchronization reconciliation is performed according to the comparison result, so as to realize the accurate data synchronization reconciliation between heterogeneous databases, avoid abnormal business operation caused by data asynchrony, and improve the reliability and stability of business processing.

[0085] The data synchronization and reconciliation system for heterogeneous databases provided in the embodiments of the present application can be configured to execute the data synchronization and reconciliation method for heterogeneous databases provided in the above embodiments, and has corresponding functions and beneficial effects.

[0086] Based on the above practical example, the present application embodiment also provides a data synchronization and reconciliation device for a heterogeneous database, referring to Figure 5 , the data synchronization and reconciliation device for heterogeneous databases includes: a processor 31, a memory 32, a communication module 33, an input device 34 and an output device 35. The memory, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the data synchronization and reconciliation method for heterogeneous databases described in any embodiment of the present application (for example, the acquisition module, query module and reconciliation module in the data synchronization and reconciliation system for heterogeneous databases). The communication module is configured to perform data transmission. The processor executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory, that is, the data synchronization and reconciliation method for heterogeneous databases described above is realized. The input device can be configured to receive input digital or character information, and generate key signal input related to the user settings and function control of the device. The output device may include a display device such as a display screen. The data synchronization and reconciliation device for heterogeneous databases provided above can be configured to execute the data synchronization and reconciliation method for heterogeneous databases provided in the above embodiment, and has corresponding functions and beneficial effects.

[0087] On the basis of the above embodiments, the embodiments of the present application further provide a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are configured to execute a data synchronization and reconciliation method for a heterogeneous database when executed by a computer processor, and the storage medium may be any of various types of memory devices or storage devices. Of course, the computer-executable instructions of the computer-readable storage medium provided in the embodiments of the present application are not limited to the data synchronization and reconciliation method for a heterogeneous database as described above, and may also execute related operations in the data synchronization and reconciliation method for a heterogeneous database provided in any embodiment of the present application.

[0088] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium, including a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the data synchronization reconciliation method of heterogeneous databases described in each embodiment of the present application.

Claims

1. A data synchronization and reconciliation method for heterogeneous databases, characterized in that: include: Acquire a data change record of the first database, the data change record including a data identifier and a version number, the first database generates the data change record when synchronizing target data to the second database, the target data including the corresponding data identifier and version number; writing the data change record into the corresponding data group according to the same data identifier, and in the case of triggering data reconciliation of the current data group, querying the second database based on the data identifier of the current data group, determining the target data matching the data identifier of the current data group, and comparing the version number of the current data group based on the version number of the matched target data; When the current data group has a version number that is the same as the version number of the target data, it is determined that the data synchronization reconciliation of the current data group is successful.

2. The data synchronization and reconciliation method for heterogeneous databases according to claim 1 is characterized in that: After comparing the version number of the current data group based on the matched version number of the target data, the method further includes: When the current data group does not have a version number identical to the version number of the target data, the corresponding target data is resynchronized from the first database to the second database.

3. The data synchronization and reconciliation method for heterogeneous databases according to claim 1 is characterized in that: After determining that the data synchronization reconciliation of the current data group is successful, the method further includes: The maximum version number of the current data group is determined, and when the version number of the target data is equal to the maximum version number, the current data group is deleted.

4. The data synchronization and reconciliation method for heterogeneous databases according to claim 1 is characterized in that: After determining that the data synchronization reconciliation of the current data group is successful, the method further includes: The maximum version number of the current data group is determined, and when the version number of the target data is smaller than the maximum version number, the data change record having a version number smaller than the version number of the target data is deleted from the current data group.

5. The data synchronization and reconciliation method for heterogeneous databases according to claim 1, characterized in that: After querying the second database based on the data identifier of the current data group, the method further includes: When it is determined that the data identifier of the current data group does not have the matching target data, and the data change rule of the data change record corresponding to the maximum version number of the current data group is deletion, it is determined that the data synchronization reconciliation of the current data group is successful, and the current data group is deleted.

6. The data synchronization and reconciliation method for heterogeneous databases according to claim 1, characterized in that: After querying the second database based on the data identifier of the current data group, the method further includes: When it is determined that the data identifier of the current data group does not have the matching target data, and the data change rule of the data change record corresponding to the maximum version number of the current data group is addition or modification, it is determined that the data synchronization reconciliation of the current data group is successful.

7. The method for data synchronization and reconciliation of heterogeneous databases according to any one of claims 1 to 6, characterized in that: In the case of triggering data reconciliation of the current data group, it also includes: The maximum version number of the current data group is monitored in real time, and when it is detected that the maximum version number has not changed within a first set period of time, the corresponding target data is resynchronized from the first database to the second database.

8. The data synchronization and reconciliation method for heterogeneous databases according to claim 7 is characterized in that: After the real-time monitoring of the maximum version number of the current data packet, the method further includes: When it is detected that the maximum version number has not changed within a second set period, a service notification is output to prompt manual repair of the target data corresponding to the first database, and the second set period is greater than the first set period.

9. A data synchronization and reconciliation system for heterogeneous databases, characterized in that: include: an acquisition module, configured to acquire a data change record of the first database, the data change record including a data identifier and a version number, the first database generating the data change record when synchronizing target data to the second database, the target data including the corresponding data identifier and version number; a query module configured to write the data change record into a corresponding data group according to the same data identifier, and, when triggering data reconciliation of the current data group, query the second database based on the data identifier of the current data group to determine the target data matching the data identifier of the current data group, and compare the version number of the current data group based on the version number of the matched target data; The reconciliation module is configured to determine that the data synchronization reconciliation of the current data group is successful when the current data group has a version number that is the same as the version number of the target data.

10. A data synchronization and reconciliation device for heterogeneous databases, characterized in that: include: memory and one or more processors; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data synchronization and reconciliation method for heterogeneous databases as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a computer processor, are configured to execute the data synchronization and reconciliation method for heterogeneous databases as described in any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product includes instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes the data synchronization and reconciliation method for heterogeneous databases as described in any one of claims 1 to 8.

Images