Method and device for data comparison and verification and data repair in data synchronization

Abstract

The application provides a data comparison and verification method and device in data synchronization, comprising: comparing the CRC check code of full data of a source end database at time t1 with the CRC check code of full data of a target end database at time t1; if the comparison result is inconsistent, determining a difference data set and generating a first data set; obtaining a transaction command set that is not executed on the difference data set in the target end database; executing the transaction command set that is not executed on the difference data set in the target end database on the difference data set in the target end database to generate a second data set; comparing the difference data set in the source end database with the second data set; if the comparison result is inconsistent, performing data repair, and the application is suitable for static data comparison scenes, dynamic data comparison scenes, isomorphic data synchronization scenes, heterogeneous data synchronization scenes, full data comparison and verification scenes and separate comparison and verification scenes, and has very wide practicality.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data comparison, verification and data repair in data synchronization. Background Art

[0002] Data comparison and verification during data synchronization can be categorized into various types. These include homogeneous and heterogeneous data comparisons based on database types, and static and dynamic data comparisons based on data change states. If the verified data is inconsistent, different methods for repair vary depending on the comparison and verification method.

[0003] Currently, the commonly used comparison methods include snapshot comparison, log comparison, control table comparison, timestamp comparison, and a combination of control table and timestamp comparison. Snapshot comparison does not rely on any mechanism and does not occupy additional system resources. However, because it is a complete copy of the data block, the databases on both ends must be of completely isomorphic database types, versions, and data structures to be compared. This is not suitable for data comparison and verification between heterogeneous databases. Log comparison: Because the operation logs of databases of different types and versions are different, this method can only be used for data comparison between databases of the same type and version, and is not suitable for data synchronization comparison and verification between heterogeneous databases. Control table comparison requires collecting all operation records of the databases on both ends, which will occupy a large amount of storage space. In addition, the comparison program will query each data item on the databases on both ends based on the received record information, which has a significant impact on the performance of the databases on both ends. Timestamp comparison: Data comparison and verification can only be completed by obtaining full data and incremental data from multiple time points and time periods, which is inefficient and has a great impact on the performance of both the source and target databases. It is not suitable for data comparison and verification of important production systems. Regarding the method combining the control table method with the timestamp: this method is to achieve synchronous incremental data acquisition, synchronization and verification by creating a temporary table to save data change information and a timestamp. This method inherits the shortcomings of the control table method and the traditional timestamp method in that they occupy too many resources. It is not suitable for comparison and verification of busy or important databases and will seriously affect the business operation of the production system. It can be seen that the above-mentioned commonly used comparison methods have not solved the following problems: 1. Efficiency issues in the data comparison process between heterogeneous databases and different database versions; 2. Rapid comparison and verification issues of full data and incremental data between heterogeneous databases; 3. Excessive resource consumption in data comparison and verification between heterogeneous databases and different database versions; 4. Accurate repair of inconsistent data without affecting the synchronization of other data. Summary of the Invention

[0004] In view of the problems in the prior art, the embodiments of the present application provide a data comparison and verification method and a data repair method in data synchronization.

[0005] In a first aspect, the embodiments of the present application provide a data comparison and verification method and a data repair method in data synchronization, comprising:

[0006] obtaining a CRC check code of full data at time t1 of a source database and a transaction serial number at the end of a transaction queue;

[0007] obtaining a CRC check code of full data at time t1 of a target database and a transaction serial number at the end of a transaction queue;

[0008] comparing the CRC check code of full data at time t1 of the source database with the CRC check code of full data at time t1 of the target database;

[0009] if the comparison result is inconsistent, determining a difference data set in the source database and a difference data set in the target database, and generating a first data set based on the difference data set in the target database;

[0010] based on the transaction serial number at the end of the transaction queue corresponding to time t1 of the source database and the transaction serial number at the end of the transaction queue corresponding to time t1 of the target database, obtaining a transaction command set that is not executed by the difference data set in the target database;

[0011] based on the transaction command set that is not executed by the difference data set in the target database, executing the difference data set in the target database to generate a second data set;

[0012] comparing the difference data set in the source database with the second data set;

[0013] if the comparison result is inconsistent, performing data repair on the inconsistent data set.

[0014] Further, the method further comprises:

[0015] comparing the CRC check code of full data at time t1 of the source database with the CRC check code of full data at time t1 of the target database, and if the comparison result is consistent, ending the verification.

[0016] Further, if the comparison result is inconsistent, the data repair on the inconsistent data set comprises:

[0017] Acquire, at the source database, a data set corresponding to the difference data set in the source database at time t2, and transfer the data set corresponding to the difference data set in the source database at time t2 to a cache in the target database;

[0018] Clear the dataset corresponding to the difference dataset at time t2 in the target database;

[0019] The dataset corresponding to the difference dataset in the source database at time t2 in the target database cache is loaded into the cleared dataset to generate a third dataset to complete the repair of the inconsistent dataset.

[0020] Furthermore, the step of loading the dataset corresponding to the difference dataset in the source database at time t2 in the target database cache into the cleared dataset to generate a third dataset to complete repairing the inconsistent dataset includes:

[0021] At time t2, an operation log analysis is performed on the data set corresponding to the difference data set in the source database at time t2 to determine an increment corresponding to the data set corresponding to the difference data set in the source database at time t2;

[0022] Based on the increment, the inconsistent data set is repaired after the third data set is generated.

[0023] In a second aspect, an embodiment of the present invention provides a device for data comparison, verification, and data repair in data synchronization, including:

[0024] The first acquisition module is used to obtain the CRC checksum of the full data at time t1 from the source database and the last transaction sequence number in the transaction queue;

[0025] The second acquisition module is used to obtain the CRC checksum of the full data at time t1 in the target database and the last transaction number in the transaction queue;

[0026] A first comparison module is configured to compare a CRC checksum of the full amount of data in the source database at time t1 with a CRC checksum of the full amount of data in the target database at time t1;

[0027] a first generating module, configured to determine a difference dataset in the source database and a difference dataset in the target database if the comparison result is inconsistent, and generate a first dataset based on the difference dataset in the target database;

[0028] a third acquisition module, configured to acquire a set of unexecuted transaction commands of the difference data set in the target database based on the last transaction sequence number in the transaction queue corresponding to time t1 of the source database and the last transaction sequence number in the transaction queue corresponding to time t1 of the target database;

[0029] A second generating module is configured to generate a second data set by executing the first data set generated by the difference data set in the target end database based on the unexecuted transaction command set in the difference data set in the target end database;

[0030] A second comparison module is used to compare the difference data set in the source database with the second data set;

[0031] The repair module is used to repair the inconsistent data sets if the comparison results are inconsistent.

[0032] Furthermore, the first comparison module is further configured to:

[0033] The CRC checksum of the full amount of data in the source database at time t1 is compared with the CRC checksum of the full amount of data in the target database at time t1. If the comparison results are consistent, the verification ends.

[0034] Furthermore, the repair module is specifically used to:

[0035] Acquire, at the source database, a data set corresponding to the difference data set in the source database at time t2, and transfer the data set corresponding to the difference data set in the source database at time t2 to a cache in the target database;

[0036] Clear the dataset corresponding to the difference dataset at time t2 in the target database;

[0037] The dataset corresponding to the difference dataset in the source database at time t2 in the target database cache is loaded into the cleared dataset to generate a third dataset to complete the repair of the inconsistent dataset.

[0038] Furthermore, when the repair module performs the step of loading the dataset corresponding to the difference dataset in the source database at time t2 in the target database cache into the cleared dataset to generate a third dataset to complete the repair of the inconsistent dataset, the repair module includes:

[0039] At time t2, an operation log analysis is performed on the data set corresponding to the difference data set in the source database at time t2 to determine an increment corresponding to the data set corresponding to the difference data set in the source database at time t2;

[0040] Based on the increment, the repair of the inconsistent data set is completed after a third data set is generated.

[0041] In a third aspect, the embodiments of the present application also provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for data comparison and data repair in data synchronization as described in the first aspect above when executing the program.

[0042] In a fourth aspect, the embodiments of the present application also provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the steps of the method for data comparison and data repair in data synchronization as described in the first aspect above.

[0043] As can be seen from the above technical solutions, the method and device for data comparison and data repair in data synchronization provided by the embodiments of the present application obtain the full data CRC check code of the source end database at time t1 and the transaction serial number at the end of the transaction queue; obtain the full data CRC check code of the target end database at time t1 and the transaction serial number at the end of the transaction queue; compare the full data CRC check code of the source end database at time t1 with the full data CRC check code of the target end database at time t1; if the comparison result is inconsistent, determine the difference data set in the source end database and the difference data set in the target end database, and generate a first data set based on the difference data set in the target end database; obtain a set of transaction commands that have not been executed by the difference data set in the target end database based on the transaction serial number at the end of the transaction queue corresponding to the source end database at time t1 and the transaction serial number at the end of the transaction queue corresponding to the target end database at time t1; execute the set of transaction commands that have not been executed by the difference data set in the target end database on the first data set generated based on the difference data set in the target end database to generate a second data set; compare the difference data set in the source end database with the second data set; if the comparison result is inconsistent, perform data repair on the inconsistent data set, thereby completing data comparison and data repair in data synchronization. The embodiments of the present application are suitable for static data comparison scenarios, dynamic data comparison scenarios, homogeneous data synchronization scenarios, heterogeneous data synchronization scenarios, full data comparison and verification scenarios, and separate comparison and verification scenarios, and have very wide practicality. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without any creative effort.

[0045] Figure 1 A flowchart of a method for data comparison verification and data repair in data synchronization according to an embodiment of the present application is shown in FIG. 1. The method includes the following steps.

[0046] Figure 2 A flowchart of a method for data comparison verification and data repair in data synchronization according to another embodiment of the present application is shown in FIG. 2. The method includes the following steps.

[0047] Figure 3 A schematic diagram of a differential transaction set according to an embodiment of the present application is shown in FIG. 3.

[0048] Figure 4 A schematic diagram of a differential transaction generation cause according to an embodiment of the present application is shown in FIG. 4.

[0049] Figure 5 A flowchart of a data repair method according to an embodiment of the present application is shown in FIG. 5. The method includes the following steps.

[0050] Figure 6 A structural schematic diagram of a device for data comparison verification and data repair in data synchronization according to an embodiment of the present application is shown in FIG. 6. The device includes the following components.

[0051] Figure 7 A schematic diagram of log analysis and transaction synthesis according to an embodiment of the present application is shown in FIG. 7.

[0052] Figure 8 A schematic diagram of an entity structure of an electronic device according to an embodiment of the present application is shown in FIG. 8. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below with reference to the drawings. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application. The method for data comparison verification and data repair in data synchronization provided by the present application will be explained and described in detail below through specific embodiments.

[0054] Figure 1 A flowchart of a method for data comparison verification and data repair in data synchronization according to an embodiment of the present application is shown in FIG. 1. The method includes the following steps. Figure 1

[0055] Step 101: Obtain the full-amount data CRC check code of a source end database at time t1 and the transaction serial number at the end of a transaction queue.

[0056] ​Step 102: Obtain the full data CRC check code of the target end database t1 moment and the transaction serial number at the end of the transaction queue.

[0057] Step 103: Compare the full data CRC check code of the source end database t1 moment and the full data CRC check code of the target end database t1 moment.

[0058] Step 104: If the comparison result is inconsistent, determine the difference data set in the source end database and the difference data set in the target end database, and generate the first data set based on the difference data set in the target end database.

[0059] Step 105: Based on the transaction serial number at the end of the transaction queue corresponding to the source end database t1 moment, and the transaction serial number at the end of the transaction queue corresponding to the target end database t1 moment, obtain the transaction command set not executed by the difference data set in the target end database.

[0060] Step 106: Based on the transaction command set not executed by the difference data set in the target end database, execute the first data set generated on the difference data set in the target end database to generate the second data set.

[0061] Step 107: Compare the difference data set in the source end database and the second data set.

[0062] Step 108: If the comparison result is inconsistent, repair the inconsistent data set.

[0063] In this embodiment, for example, refer to Figure 2, obtain the CRC check code of the full data of the source database at time t1 and the last completed transaction sequence number S1 (that is, the source database, the last transaction sequence number in the transaction queue), obtain the CRC check code of the full data of the target database at time t1 and the last completed transaction sequence number S1' (that is, the target database, the last transaction sequence number in the transaction queue); compare the full CRC check codes of the source database and the target database at time t1 (that is, compare the CRC check code of the full data of the source database at time t1 with the CRC check code of the full data of the target database at time t1). If the data are consistent, it means that no incremental data is generated during this period, and the verification is completed; if the data are inconsistent, according to the comparison result, obtain the source database difference data set (t1, a1), obtain the target database difference data set (t1, b1), and load (t1, b1) into the verification server to generate the data set (t1, b1'), that is, determine the difference data set in the source database and the difference data set in the target database, and based on the difference data set in the target database a first data set is generated from the difference data set; a set of transaction commands that have not been executed for the difference data (t1, b1) is obtained based on the last transaction number S1 at the source end and the last transaction number S1' at the target end (i.e., a set of transaction commands that have not been executed for the difference data set in the target database is obtained based on the last transaction sequence number in the transaction queue corresponding to the source database at time t1 and the last transaction sequence number in the transaction queue corresponding to the target database at time t1), and the set of transaction commands that have not been executed for the difference data set in the target database is executed on the data set (t1, b1') to generate a data set (t1, b2), i.e., a second data set is generated based on the first data set generated from the difference data set in the target database based on the set of transaction commands that have not been executed for the difference data set in the target database; the data sets (t1, a1) and (t1, b2) are compared for consistency (i.e., the difference data set in the source database and the second data set are compared); if the comparison and verification results are inconsistent, it means that there are indeed inconsistent data in the source and target databases, and data repair is performed on the inconsistent data sets.

[0064] In this embodiment, it should be noted that, in the process of data synchronization, the embodiment of the present invention refers to the data sending end as the source database and the data receiving end as the target database; obtains the CRC check value of the full data of the source database at time t1, wherein time t1 refers to any time when data comparison and verification begins, and is not required to have some special meaning. The full data can refer to the entire database data set or a collection of certain data tables; obtains the last transaction sequence number S1 completed by the full data of the source at time t1, wherein the transaction sequence number refers to the sequence of transactions written by the database receiving application. Different databases have different ways of obtaining it, such as Oracle database is obtained in redolog, MySQL is obtained in binlog, but the actual meaning is the same; obtains the CRC check value of the full data of the target database at time t1 value; obtain the last transaction sequence number S1' of the target-side full data completed at time t1; compare the full data of the source and target databases at time t1; if the verification results are consistent, it means that there is no data synchronization delay during this period and the data verification is complete; if the verification results are inconsistent, based on the full data verification results: obtain the data set (t1, a1) corresponding to the inconsistent data at time t1 in the source database, and obtain the data set (t1, b1) corresponding to the inconsistent data at time t1 in the target database; in order not to affect the formal system in the data synchronization environment, it is necessary to transfer and load the b1 data set of the target database to the verification server to generate the data set (t1, b1'); according to the source database log analysis module, obtain all transaction sets (S1', S1) between the source database transaction sequence number S1 and the transaction sequence number S1'; see Figure 3 The difference transaction set diagram shown in the figure shows that the transaction set (S1', S1) is executed on the data verification server b1', and the data set (t1, b2) is obtained after completion; see Figure 4 The diagram below shows the cause of differential trading. It compares the two data sets (t1, a1) and (t1, b2). If the comparison results of (t1, a1) and (t1, b2) are consistent, the full and incremental data comparison and verification are completed. If the comparison results of (t1, a1) and (t1, b2) are inconsistent, it is necessary to repair the data set corresponding to b1 in the target database based on the content of the data set corresponding to a1 in the source database.

[0065] In the embodiment of the present invention, it should be noted that the embodiment of the present invention is different from the snapshot comparison method. For example, the snapshot comparison method is to "take a picture" of the data in the databases at both ends at a certain moment, generate two static files describing the instantaneous state of the data in the databases, and then compare and verify the two snapshots. This method is more suitable for scenarios with the same database type and version, but is not suitable for comparison and verification of heterogeneous databases. This is because the storage methods and data type representation methods of heterogeneous databases may be different, which results in snapshot differences regardless of whether the data is consistent. In this embodiment, CRC verification is performed on the data content, which shields the differences in database types and database versions and is suitable for The scope is wider; the embodiment of the present invention is relative to the log comparison method, such as the log comparison method is to capture the changes of the copy object by analyzing the database log information, and then compare the log change information of the databases at both ends to verify the consistency of the data. The defect of this comparison method is similar to the snapshot comparison method, which is only applicable to the data comparison of isomorphic data synchronization, that is, the difference in database type and version will cause the log itself and the log analysis result to be different, and this embodiment uses CRC check and differential data completion method to perform data comparison and verification, which is applicable to the full and incremental comparison of isomorphic databases and heterogeneous databases; the embodiment of the present invention is relative to the control table comparison method, such as the control table comparison method is to create a control table for each source table to be synchronized, and the control table is only Including the primary key field of the source table and some control information fields (update time, update type, etc.). When a record in the source table changes, the corresponding primary key record and time information of the control table are added or modified at the same time. Then, the corresponding updated content of the databases on both ends is checked back for data comparison and verification. The main drawback of this comparison method is that it frequently queries data on the source database and the target database, which will occupy a large amount of computing resources and affect the normal operation of the business when the system is busy. In this embodiment, only one full data CRC check and one differential data CRC check are required, which greatly reduces the resource consumption of the source database and the target database. The differential data completion is performed in the comparison and verification server, and will not occupy the production system. Resources; Compared with the timestamp method, the embodiment of the present invention requires a timestamp field in each table in the relevant application system to record the modification time of each table, and then performs data comparison and verification based on the data obtained in different time periods and time points. The main drawback of this solution is that it is necessary to obtain the full data and incremental data of the source and target ends multiple times to complete a synchronous data comparison and verification. The process has too many links and is inefficient. It occupies a large amount of I / O resources and computing resources of the source and target databases. However, this embodiment uses a differential transaction completion method to obtain the full data CRC check code and differential transaction information at one time to complete the data comparison and verification, and ensure the convergence and integrity of the data verification;Compared with the control table and timestamp combined comparison method, the control table and timestamp combined comparison method is a method for realizing synchronization of incremental data checking based on the combination of control table (creating a temporary table to save data change information) and timestamp in the data synchronization process. The method also inherits the defects of the control table method and the timestamp method, that is, the comparison and checking process is too long, and the system resources are occupied too high. The embodiment is a method for completing the differential transaction. The CRC check code of the full data and the differential transaction information are obtained at one time to complete the data comparison and checking, and the convergence and integrity of the data checking are ensured. It can be seen that the data comparison and checking and data repair method in the data synchronization provided by the embodiment can achieve accurate repair of inconsistent data, and does not affect the synchronization function of other normal data in the repair process. That is, the embodiment is suitable for static data comparison scene, dynamic data comparison scene, homogeneous data synchronization scene, heterogeneous data synchronization scene, full data comparison and checking scene, and separate comparison and checking scene of part of data table, and has very wide practicality. The embodiment adopts a differential data completion method for inconsistent data sets to perform incremental data comparison, shortens the whole data comparison and checking process, and greatly reduces the resource occupation and performance influence on the source end database and the target end database. Whether it is full data checking or incremental data checking, it needs to be realized through the content check code. The traditional method is to perform data comparison and checking according to the primary key ID, timestamp and other information. In the embodiment, for the table with primary key, the CRC check code is generated based on the primary key ID and timestamp, for the table without primary key, the CRC column check code is generated based on the column of the data table, and the check codes of different data sets can be compared. In this way, the data comparison and checking range of the table with primary key and the table without primary key is completely covered.

[0066] As can be seen from the above technical solution, the method for data comparison, verification and data repair in data synchronization provided by the embodiment of the present invention obtains the CRC check code of the full data of the source database at time t1 and the transaction sequence number at the end of the transaction queue; obtains the CRC check code of the full data of the target database at time t1 and the transaction sequence number at the end of the transaction queue; compares the CRC check code of the full data of the source database at time t1 with the CRC check code of the full data of the target database at time t1; if the comparison results are inconsistent, determines the difference data set in the source database and the difference data set in the target database, and generates a first data set based on the difference data set in the target database; based on the transaction sequence number at the end of the transaction queue corresponding to the source database at time t1, and, with The target-side database uses the last transaction number in the transaction queue corresponding to time t1 to obtain a set of unexecuted transaction commands for the differential data set in the target-side database; generates a second data set based on the first data set generated by the differential data set in the target-side database based on the set of unexecuted transaction commands for the differential data set in the target-side database; compares the differential data set in the source-side database with the second data set; if the comparison results are inconsistent, repairs the inconsistent data sets, thereby completing data comparison and repair in data synchronization. The embodiments of the present invention are applicable to static data comparison scenarios, dynamic data comparison scenarios, homogeneous data synchronization scenarios, heterogeneous data synchronization scenarios, full data comparison and verification scenarios, and separate comparison and verification scenarios, and have very wide practicality.

[0067] Based on the above embodiment, this embodiment further includes:

[0068] The CRC checksum of the full amount of data in the source database at time t1 is compared with the CRC checksum of the full amount of data in the target database at time t1. If the comparison results are consistent, the verification ends.

[0069] Based on the above embodiment, in this embodiment, if the comparison results are inconsistent, data repair is performed on the inconsistent data set, including:

[0070] Acquire, at the source database, a data set corresponding to the difference data set in the source database at time t2, and transfer the data set corresponding to the difference data set in the source database at time t2 to a cache in the target database;

[0071] Clear the dataset corresponding to the difference dataset at time t2 in the target database;

[0072] The dataset corresponding to the difference dataset in the source database at time t2 in the target database cache is loaded into the cleared dataset to generate a third dataset to complete the repair of the inconsistent dataset.

[0073] In the embodiment, for example, referring to Figure 5 , at t2, the corresponding data set (t2, a1) of a1 in the source database is acquired; the data set (t2, a1) is transmitted to the cache of the target database (that is, the corresponding data set of the differential data set in the source database at t2 is acquired in the source database, and the corresponding data set of the differential data set in the source database at t2 is transmitted to the cache of the target database); the corresponding data set of b1 in the target database at t2 is emptied, that is, the corresponding data set of the differential data set in the target database at t2 is emptied, and it should be noted that the emptying refers to emptying the content of the table, and the data table structure is to be retained; the (t2, a1) in the cache of the target database is loaded into the emptied data set, and (t2, b1) is generated to complete the repair of the inconsistent data set, that is, the corresponding data set of the differential data set in the source database in the cache of the target database is loaded into the emptied data set, and the third data set is generated to complete the repair of the inconsistent data set.

[0074] From the above technical solution, it can be known that the method for data comparison and verification and data repair in data synchronization provided by the embodiment of the application does not affect the synchronization of other data sets in the database in the repair process, can perform pinpoint repair on the inconsistent data set, and does not need to perform full-database repair or full-data set repair.

[0075] On the basis of the above embodiment, in the embodiment, the loading of the corresponding data set of the differential data set in the source database in the cache of the target database into the emptied data set to generate the third data set to complete the repair of the inconsistent data set comprises:

[0076] At t2, operation log analysis is performed on the corresponding data set of the differential data set in the source database at t2, and the increment corresponding to the corresponding data set of the differential data set in the source database at t2 is determined.

[0077] Based on the increment, the repair of the inconsistent data set is completed after the third data set is generated.

[0078] In the embodiment, for example, at t2, the transaction log analysis device starts operation log analysis on the data set (t2, a1), and continuously sends the analysis result to the target database; the received increment transaction for (t2, a1) is put into the cache area in the target database, and the transaction in the cache is continuously loaded into the database after the data set (t2, b1) is generated, so that the precise repair of the inconsistent data set is completed, and this process does not affect the data synchronization work of other verified data sets.

[0079] From the above technical solutions, the method for data comparison and verification and data repair in data synchronization provided by the embodiments of the present application has two features, the first is the ability of point fixing and small range data repair, and the second is that the data repair process does not affect the normal data synchronization function.

[0080] Figure 6 The structural diagram of the device for data comparison and verification and data repair in data synchronization provided by an embodiment of the present application is shown in FIG. 1, which comprises a first acquisition module 201, a second acquisition module 202, a first comparison module 203, a first generation module 204, a third acquisition module 205, a second generation module 206, a second comparison module 207 and a repair module 208. Figure 6

[0081] The first acquisition module 201 is configured to acquire the full data CRC check code of the source end database at time t1 and the transaction serial number at the end of the transaction queue.

[0082] The second acquisition module 202 is configured to acquire the full data CRC check code of the target end database at time t1 and the transaction serial number at the end of the transaction queue.

[0083] The first comparison module 203 is configured to compare the full data CRC check code of the source end database at time t1 with the full data CRC check code of the target end database at time t1.

[0084] The first generation module 204 is configured to determine the difference data set in the source end database and the difference data set in the target end database if the comparison result is inconsistent, and generate a first data set based on the difference data set in the target end database.

[0085] The third acquisition module 205 is configured to acquire the transaction command set that has not been executed by the difference data set in the target end database based on the transaction serial number at the end of the transaction queue corresponding to the source end database at time t1 and the transaction serial number at the end of the transaction queue corresponding to the target end database at time t1.

[0086] The second generation module 206 is configured to execute the transaction command set that has not been executed by the difference data set in the target end database on the first data set generated based on the difference data set in the target end database to generate a second data set.

[0087] The second comparison module 207 is configured to compare the difference data set in the source end database with the second data set.

[0088] The repair module 208 is configured to perform data repair on the inconsistent data set if the comparison result is inconsistent. ​

[0089] Based on the above embodiment, it is preferred to further include: a full data acquisition module for acquiring the full data of a certain data set at a certain moment; a CRC generation and verification module for generating a CRC check code for a certain data set. For a table with a primary key, the CRC check code is generated based on the primary key ID and timestamp; for a table without a primary key, the CRC column check code is generated based on the column of the data table, and the check codes of different data sets can be compared; a database log analysis module for users to analyze the online logs of the source database (such as Oracle's redolog, MySQL's binlog, etc.) and obtain information such as SQL transactions, transaction sequence numbers, and transaction times. Figure 7 The log analysis and transaction synthesis diagram shown in the figure; the transaction loading module is a device that can load the SQL transaction data obtained by analyzing the source database into the target database or a collection of certain data tables in the order of transaction numbers; the transaction loading module is a device that can load the SQL transaction data obtained by analyzing the source database into the target database or a collection of certain data tables in the order of transaction numbers; the data clearing module retains the data table structure by clearing the data content of the specified data set table; the data loading module loads the data obtained by the full data acquisition module into the specified table structure; the database log analysis module allows users to analyze the source database online log (such as Oracle's redolog, MySQL's binlog, etc.) and obtain information such as SQL transactions, transaction numbers, and transaction times.

[0090] The device for data comparison, verification and data repair in data synchronization provided by the embodiment of the present invention can be specifically used to execute the method for data comparison, verification and data repair in data synchronization in the above embodiment. Its technical principles and beneficial effects are similar. Please refer to the above embodiment for details and will not be repeated here.

[0091] Based on the same inventive concept, an embodiment of the present invention provides an electronic device, see Figure 8 , the electronic device specifically includes the following: a processor 301, a communication interface 303, a memory 302 and a communication bus 304;

[0092] Among them, the processor 301, the communication interface 303, and the memory 302 complete mutual communication through the communication bus 304; the communication interface 303 is used to realize information transmission between various modeling software and intelligent manufacturing equipment module libraries and other related equipment; the processor 301 is used to call the computer program in the memory 302, and the processor implements the methods provided by the above-mentioned method embodiments when executing the computer program. For example, when the processor executes the computer program, it implements the following steps: obtaining the CRC check code of the full data of the source database at time t1 and the transaction sequence number at the end of the transaction queue; obtaining the CRC check code of the full data of the target database at time t1 and the transaction sequence number at the end of the transaction queue; checking the CRC check code of the full data of the source database at time t1 and the CRC check code of the full data of the target database at time t1 The method comprises the steps of: comparing the difference data set in the source database and the difference data set in the target database; if the comparison result is inconsistent, determining the difference data set in the source database and the difference data set in the target database, and generating a first data set based on the difference data set in the target database; obtaining a set of unexecuted transaction commands of the difference data set in the target database based on the transaction sequence number at the end of the transaction queue corresponding to time t1 of the source database and the transaction sequence number at the end of the transaction queue corresponding to time t1 of the target database; executing the unexecuted transaction commands of the difference data set in the target database on the first data set generated from the difference data set in the target database to generate a second data set; comparing the difference data set in the source database and the second data set; and performing data repair on the inconsistent data sets if the comparison result is inconsistent.

[0093] Based on the same inventive concept, another embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the method provided by the above-mentioned method embodiments, for example, obtaining the CRC checksum of the full data of the source database at time t1 and the transaction sequence number at the end of the transaction queue; obtaining the CRC checksum of the full data of the target database at time t1 and the transaction sequence number at the end of the transaction queue; comparing the CRC checksum of the full data of the source database at time t1 with the CRC checksum of the full data of the target database at time t1; if the comparison results are inconsistent, determining the difference between the source database and the target database. The method comprises the steps of: obtaining a differential data set in the source database and generating a first data set based on the differential data set in the target database; obtaining a set of unexecuted transaction commands of the differential data set in the target database based on the transaction sequence number at the end of the transaction queue corresponding to time t1 of the source database and the transaction sequence number at the end of the transaction queue corresponding to time t1 of the target database; generating a second data set based on the first data set generated from the differential data set in the target database based on the transaction sequence number at the end of the transaction queue corresponding to time t1 of the target database; comparing the differential data set in the source database and the second data set; and repairing the inconsistent data sets if the comparison results are inconsistent.

[0094] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. That is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0095] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or certain parts of the embodiment.

[0096] Furthermore, in the present invention, terms such as "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of the present invention, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0097] In addition, in the present invention, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further limitations, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0098] In addition, in the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for data comparison, verification and data repair in data synchronization, characterized in that: include: Obtain the CRC checksum of the full data at time t1 in the source database and the last transaction number in the transaction queue; Obtain the CRC checksum of the full data at time t1 in the target database and the last transaction number in the transaction queue; Compare the CRC checksum of the full amount of data in the source database at time t1 with the CRC checksum of the full amount of data in the target database at time t1; If the comparison result is inconsistent, determining the difference data set in the source database and the difference data set in the target database, and generating a first data set based on the difference data set in the target database; Based on the last transaction sequence number in the transaction queue corresponding to time t1 of the source database and the last transaction sequence number in the transaction queue corresponding to time t1 of the target database, obtaining a set of unexecuted transaction commands of the difference data set in the target database; generating a second data set by executing a set of unexecuted transaction commands in the difference data set in the target-side database on the first data set generated from the difference data set in the target-side database; Comparing the difference dataset in the source database with the second dataset; If the comparison results are inconsistent, data repair is performed on the inconsistent data set.

2. The method for data comparison, verification and data repair in data synchronization according to claim 1, characterized in that: Also includes: The CRC checksum of the full amount of data in the source database at time t1 is compared with the CRC checksum of the full amount of data in the target database at time t1. If the comparison results are consistent, the verification ends.

3. The method for data comparison, verification and data repair in data synchronization according to claim 1, characterized in that: If the comparison results are inconsistent, data repair is performed on the inconsistent data set, including: Acquire, at the source database, a data set corresponding to the difference data set in the source database at time t2, and transfer the data set corresponding to the difference data set in the source database at time t2 to a cache in the target database; Clear the dataset corresponding to the difference dataset at time t2 in the target database; The dataset corresponding to the difference dataset in the source database at time t2 in the target database cache is loaded into the cleared dataset to generate a third dataset to complete the repair of the inconsistent dataset.

4. The method for data comparison, verification and data repair in data synchronization according to claim 3, characterized in that: The step of loading the dataset corresponding to the difference dataset in the source database at time t2 in the target database cache into the cleared dataset to generate a third dataset to complete repairing the inconsistent dataset includes: At time t2, an operation log analysis is performed on the data set corresponding to the difference data set in the source database at time t2 to determine an increment corresponding to the data set corresponding to the difference data set in the source database at time t2; Based on the increment, the inconsistent data set is repaired after the third data set is generated.

5. A device for data comparison, verification and data repair in data synchronization, characterized in that: include: The first acquisition module is used to obtain the CRC checksum of the full data at time t1 from the source database and the last transaction number in the transaction queue; The second acquisition module is used to obtain the CRC checksum of the full data at time t1 in the target database and the last transaction number in the transaction queue; A first comparison module is configured to compare a CRC checksum of the full amount of data in the source database at time t1 with a CRC checksum of the full amount of data in the target database at time t1; a first generating module, configured to determine a difference dataset in the source database and a difference dataset in the target database if the comparison result is inconsistent, and generate a first dataset based on the difference dataset in the target database; a third acquisition module, configured to acquire a set of unexecuted transaction commands of the difference data set in the target database based on the last transaction sequence number in the transaction queue corresponding to time t1 of the source database and the last transaction sequence number in the transaction queue corresponding to time t1 of the target database; A second generating module is configured to generate a second data set by executing the first data set generated by the difference data set in the target end database based on the unexecuted transaction command set in the difference data set in the target end database; A second comparison module is used to compare the difference data set in the source database with the second data set; The repair module is used to repair the inconsistent data sets if the comparison results are inconsistent.

6. The device for data comparison, verification and data repair in data synchronization according to claim 5, characterized in that: The first comparison module is further used to: The CRC checksum of the full amount of data in the source database at time t1 is compared with the CRC checksum of the full amount of data in the target database at time t1. If the comparison results are consistent, the verification ends.

7. The device for data comparison, verification and data repair in data synchronization according to claim 5, characterized in that: The repair module is specifically used to: Acquire, at the source database, a data set corresponding to the difference data set in the source database at time t2, and transfer the data set corresponding to the difference data set in the source database at time t2 to a cache in the target database; Clear the dataset corresponding to the difference dataset at time t2 in the target database; The dataset corresponding to the difference dataset in the source database at time t2 in the target database cache is loaded into the cleared dataset to generate a third dataset to complete the repair of the inconsistent dataset.

8. The device for data comparison, verification and data repair in data synchronization according to claim 7, characterized in that: When the repair module performs the step of loading the dataset corresponding to the difference dataset in the source database at time t2 in the target database cache into the cleared dataset to generate a third dataset to complete the repair of the inconsistent dataset, the method includes: At time t2, an operation log analysis is performed on the data set corresponding to the difference data set in the source database at time t2 to determine an increment corresponding to the data set corresponding to the difference data set in the source database at time t2; Based on the increment, the inconsistent data set is repaired after the third data set is generated.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for data comparison, verification and data repair in data synchronization according to any one of claims 1 to 4 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for data comparison, verification and data repair in data synchronization as claimed in any one of claims 1 to 4 is implemented.

Images