A data consistency comparison method and apparatus

Abstract

The application relates to the technical field of computers, and provides a data consistency comparison method and device.The method comprises the following steps: comparing a first data copy with a corresponding first historical data copy to obtain a first modification record set; comparing a second data copy with a corresponding second historical data copy to obtain a second modification record set; and obtaining a data comparison result between the first data copy and the second data copy according to a historical comparison result, the first modification record set and the second modification record set.When the data copy is compared with the historical data copy of the data copy, the data which is not modified can be directly filtered, the data does not occupy resources in the subsequent comparison process, the resource occupation of the data consistency comparison is reduced, and the comparison efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data consistency comparison method and apparatus. Background Technology

[0002] With the rapid development of information technology infrastructure, business processes are becoming increasingly complex, and the coupling between different business units is becoming stronger. Often, different business units need to access the same data. In this situation, the data accessed by various business units becomes "hot data," and competition for data access can cause some business units to wait, leading to a decline in business performance. To solve this problem, the common approach is to create multiple copies of the data, with each business unit accessing a different copy. These copies are periodically synchronized to ensure data consistency across different business units. Typically, when synchronizing large amounts of data, direct data synchronization is resource-intensive. Therefore, a data consistency check is performed first, and synchronization is only performed on inconsistent data. In this case, the data consistency across all copies becomes particularly important.

[0003] Data comparison is an important method for data consistency checking. The existing technology involves directly comparing data from replicas. For example, comparing data between replica A and replica B typically involves three steps: 1. Obtain data from replica A and calculate the MD5 value of each row; 2. Obtain data from replica B and calculate the MD5 value of each row; 3. Compare the MD5 values ​​between replica A and replica B. If there is a difference in the MD5 values, then there is data inconsistency between replica A and replica B.

[0004] The above data comparison methods suffer from numerous invalid comparisons when the table data volume is large and repeated comparisons are required. For example, when the system needs to compare replicas A and B daily, each comparison is a full comparison, which is time-consuming if the data volume in the replicas is large. Furthermore, in large-scale data storage management, each business does not need to modify every single piece of data. More often, over time, a large amount of static data accumulates, which is unlikely to be modified and may account for 50% or more of the total data volume. In data storage that accumulates over time, historical data is typically no longer modified after a certain period, such as data from several months or years ago. In this case, a significant portion of the time spent on comparing historical data during full data comparison not only reduces the efficiency of data consistency comparison but also consumes substantial resources.

[0005] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0006] The technical problem that this invention aims to solve is that existing data consistency comparison methods consume a lot of resources in comparing historical data without modification, resulting in high resource consumption and low efficiency.

[0007] In a first aspect, the present invention provides a data consistency comparison method, comprising:

[0008] The first data copy is compared with the corresponding first historical data copy to obtain the first set of modification records;

[0009] The second data copy is compared with the corresponding second historical data copy to obtain the second set of modification records;

[0010] Based on the historical comparison results, the first modification record set, and the second modification record set, the data comparison results between the first data copy and the second data copy are obtained;

[0011] In this process, the data copy used in the previous comparison is used as the historical data copy for the next comparison, and the data comparison result obtained in the previous comparison is used as the historical comparison result for the next comparison. In the first comparison, the first data copy is compared with the second data copy to obtain the data comparison result.

[0012] Preferably, the step of comparing the first data copy with the second data copy to obtain a data comparison result specifically includes:

[0013] Compare the first packet in the first data copy with the second packet in the second data copy;

[0014] If the comparison shows that the first group and the second group are inconsistent, the first data in the first group and the second data in the second group are compared one by one to determine the first and second data that are inconsistent with each other. In this process, the data in the data copy is divided into multiple groups, each group containing multiple data entries. When dividing the data into groups, the data correspondence between different data copies is used to make the groups obtained from different data copies correspond to each other.

[0015] Based on the inconsistencies between the first and second data, corresponding comparison difference records are generated and stored in the data comparison results.

[0016] Preferably, comparing the first group in the first data copy with the second group in the second data copy specifically includes:

[0017] Compare the group check codes of the first group with those of the second group to see if they match; wherein, based on generating data check codes for each data item in the corresponding group, the group check code of the group is generated according to the data check codes of all data in the group.

[0018] Preferably, the step of comparing the first data copy with the corresponding first historical data copy to obtain the first modification record set specifically includes:

[0019] Compare the third group in the first data copy with the fourth group in the first historical data copy;

[0020] If the comparison shows that the third group and the fourth group are inconsistent, then the third data in the third group and the fourth data in the fourth group are compared one by one. If the comparison shows that the third data and the fourth data are inconsistent, then a corresponding modification record is generated based on the third data and the fourth data and stored in the first modification record set.

[0021] Preferably, the step of generating corresponding modification records based on the third data and the fourth data specifically includes:

[0022] Using the fourth data as the data before modification and the third data as the data after modification, the data check code of the data before modification and the data check code of the data after modification are used as a modification record.

[0023] Preferably, comparing the third group in the first data copy with the fourth group in the first historical data copy specifically includes:

[0024] The group check code of the third group is compared with the group check code of the fourth group to determine if they are consistent. The third group, which was used to divide the first data copy during the first comparison, is reused, and the group check code of the third group generated during the previous comparison is used as the group check code of the fourth group in the historical data copy for the next comparison.

[0025] Preferably, obtaining the data comparison result between the first data copy and the second data copy based on the historical comparison results, the first modification record set, and the second modification record set specifically includes:

[0026] The first and second modified record sets are deduplicated to remove duplicate modified records from both sets.

[0027] The modified records in the first and second modified record sets after deduplication are taken as target modified records. The target modified records are compared with the historical comparison results to determine whether there are any comparison difference records in the historical comparison results that match the target modified records.

[0028] If it exists, the comparison difference record is deleted from the historical comparison results; if it does not exist, a corresponding comparison difference record is generated according to the target modification record, and the comparison difference record is added to the historical comparison results, so that the final historical comparison results are used as the data comparison results between the first data copy and the second data copy.

[0029] Preferably, determining whether there is a comparison difference record in the historical comparison results that matches the target modification record specifically includes:

[0030] When the target modified record belongs to the first modified record set, it is determined whether the first data in the comparison difference record is consistent with the data before the modification of the target modified record, and whether the second data in the comparison difference record is consistent with the data after the modification of the target modified record. If both are consistent, then there is a comparison difference record that matches the target modified record in the historical comparison results.

[0031] When the target modified record belongs to the second modified record set, it is determined whether the second data in the comparison difference record is consistent with the data before modification of the target modified record, and whether the first data in the comparison difference record is consistent with the data after modification of the target modified record. If both are consistent, then there is a comparison difference record that matches the target modified record in the historical comparison results.

[0032] Preferably, the step of generating a corresponding comparison difference record based on the target modification record specifically includes:

[0033] When the target modified record belongs to the first modified record set, the modified data of the target modified record is used as the first data, the corresponding second data is searched, and a comparison difference record is generated.

[0034] When the target modified record belongs to the second modified record set, the modified data of the target modified record is used as the second data, the corresponding first data is searched, and a comparison difference record is generated.

[0035] Secondly, the present invention also provides a data consistency comparison device for implementing the data consistency comparison method described in the first aspect, the device comprising:

[0036] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the data consistency comparison method described in the first aspect.

[0037] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions, which are executed by one or more processors to perform the data consistency comparison method described in the first aspect.

[0038] This invention obtains modification records by comparing a data copy with its own historical data copy, and then combines the results with the historical comparison results to obtain the final comparison result. This allows for the direct filtering of data copies that have not been modified when comparing them with their own historical data copies, so that they do not consume resources in subsequent comparison processes, thereby reducing the resource consumption of data consistency comparison and improving comparison efficiency. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0040] Figure 1 This is a flowchart illustrating a data consistency comparison method provided in an embodiment of the present invention;

[0041] Figure 2 This is a flowchart illustrating a data consistency comparison method provided in an embodiment of the present invention;

[0042] Figure 3 This is a flowchart illustrating a data consistency comparison method provided in an embodiment of the present invention;

[0043] Figure 4 This is a flowchart illustrating a data consistency comparison method provided in an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram illustrating an application scenario of a data consistency comparison method provided in an embodiment of the present invention;

[0045] Figure 6 This is a flowchart illustrating a data consistency comparison method provided in an embodiment of the present invention;

[0046] Figure 7 This is a flowchart illustrating a data consistency comparison method provided in an embodiment of the present invention;

[0047] Figure 8 This is a schematic diagram of a data consistency comparison method provided in an embodiment of the present invention;

[0048] Figure 9This is a schematic diagram of a data consistency comparison method provided in an embodiment of the present invention;

[0049] Figure 10 This is a schematic diagram of a data consistency comparison method provided in an embodiment of the present invention;

[0050] Figure 11 This is a schematic diagram of a data consistency comparison method provided in an embodiment of the present invention;

[0051] Figure 12 This is a schematic diagram of a data consistency comparison method provided in an embodiment of the present invention;

[0052] Figure 13 This is a schematic diagram of a data consistency comparison method provided in an embodiment of the present invention;

[0053] Figure 14 This is a flowchart illustrating a data consistency comparison method provided in an embodiment of the present invention;

[0054] Figure 15 This is a schematic diagram of a data consistency comparison method provided in an embodiment of the present invention;

[0055] Figure 16 This is a schematic diagram of a data consistency comparison method provided in an embodiment of the present invention;

[0056] Figure 17 This is a schematic diagram of a data consistency comparison method provided in an embodiment of the present invention;

[0057] Figure 18 This is a schematic diagram of the architecture of a data consistency comparison device provided in an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0059] To make the application scenarios and technical advantages of this invention clearer, the application scenarios of this invention will be described in detail below. This invention is applicable to data consistency comparison scenarios with a large amount of historical data that does not need modification. In this scenario, this invention can demonstrate performance advantages that are significantly different from existing technologies. While this invention is also applicable to other scenarios, its performance advantages are only demonstrated when the preferred implementation of this invention is adopted. Several scenarios to which this invention is applicable will be specifically illustrated below with examples.

[0060] The first scenario to which this invention applies is when multiple systems use the same set of data. Due to the performance limitations of the database, when the data volume is large and the frequency of data modification by multiple systems is high, if multiple systems directly use the database, it may lead to a decrease in system performance. Therefore, it is common practice to generate multiple data copies from the database, with each copy used by one system, and to synchronize the data copies periodically. In this case, over time, historical data that does not need to be modified is usually generated. In this scenario, using this invention can greatly improve the efficiency of data comparison and synchronization and reduce its resource consumption.

[0061] Another application scenario for which this invention is applicable is data backup. For example, in order to prevent the system from being affected when the corresponding database stops working, one or more backup databases are set up to synchronize data between the primary database and the backup database. Similarly, in scenarios where historical data that does not need to be modified accumulates over time, this invention can give full play to its advantages.

[0062] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0063] Example 1:

[0064] Embodiment 1 of the present invention provides a data consistency comparison method, such as... Figure 1 As shown, it includes:

[0065] In step 201, the first data copy is compared with the corresponding first historical data copy to obtain the first modification record set.

[0066] In step 202, the second data copy is compared with the corresponding second historical data copy to obtain the second modification record set.

[0067] In step 203, the data comparison result between the first data copy and the second data copy is obtained based on the historical comparison results, the first modified record set, and the second modified record set.

[0068] In this process, the data copy used in the previous comparison is used as the historical data copy for the next comparison, and the data comparison result obtained in the previous comparison is used as the historical comparison result for the next comparison. In the first comparison, the first data copy is compared with the second data copy to obtain the data comparison result.

[0069] The first and second data copies can be obtained from the same source data file, and may contain part or all of the data in the source data file. Alternatively, the second data copy may be obtained from the first data copy, and the second data copy may contain all or part of the data in the first data copy. Furthermore, during subsequent operation, the first and second data copies may be modified by different business accesses, leading to asynchronous changes to the first and second data copies. That is, the values ​​of the data in the first data copy may differ from the values ​​of the data in the second data copy. Therefore, the data consistency comparison method described in this embodiment is used to determine the discrepancy before synchronizing the data between the first and second data copies.

[0070] The comparison can be a direct comparison of the data in the first data copy and the second data copy, or it can be a comparison using a checksum.

[0071] The first modification record set is the collection of data modification records generated by the first data copy during the period between the next comparison and the previous comparison. The first historical data copy is the first data copy at the time of the previous comparison. The second modification record set is the collection of data modification records generated by the second data copy during the period between the next comparison and the previous comparison. The second historical data copy is the first data copy at the time of the previous comparison.

[0072] It should be noted that the "previous comparison" and "next comparison" mentioned in this embodiment refer to two adjacent comparison processes. For example, if three comparisons have been performed up to a certain time, for ease of description, these three comparisons are referred to in chronological order as: the first comparison, the second comparison, and the third comparison. The first comparison is the "previous comparison" of the second comparison, the second comparison is the "next comparison" of the first comparison, the second comparison is the "previous comparison" of the third comparison, and the third comparison is the "next comparison" of the second comparison.

[0073] This embodiment compares the data copy with its own historical data copy to obtain the modification records, and then combines the results with the historical comparison results to obtain the final comparison result. This allows for the direct filtering of data copies that have not been modified when comparing them with their own historical data copies, so that they do not consume resources in subsequent comparison processes. This reduces the resource consumption of data consistency comparison and improves comparison efficiency.

[0074] In practical use, the first data copy and the second data copy may not be completely identical. It is necessary to first confirm the data differences between the first data copy and the second data copy, and then use the data differences as a benchmark. By combining the first modification record set and the second modification record set, the differences between the first data copy and the second data copy can be continuously iterated and updated. The process of the first comparison between the first data copy and the second data copy is described below.

[0075] Since the data volume of the first and second data copies may be very large, in order to further improve the efficiency of data consistency comparison, in conjunction with the above embodiments, there is the following preferred embodiment, namely, comparing the first and second data copies to obtain the data comparison result, such as... Figure 2 As shown, it specifically includes:

[0076] In step 301, the first group in the first data copy is compared with the second group in the second data copy.

[0077] In step 302, if the comparison shows that the first group and the second group are inconsistent, the first data in the first group and the second data in the second group are compared one by one to determine the first and second data that are inconsistent with each other. In this process, the data in the data copy is divided into multiple groups, and each group contains multiple data. When dividing the data into groups, the groups obtained from different data copies are made to correspond to each other according to the data correspondence between each data copy.

[0078] In step 303, a corresponding comparison difference record is generated based on the inconsistency between the first and second data, and the comparison difference record is stored in the data comparison result.

[0079] In this embodiment, the first data copy and the second data copy are grouped using the same grouping method. Specifically, the first data copy is grouped to obtain at least one first group, wherein each first group contains multiple first data entries. The second data copy is grouped to obtain at least one second group, wherein each second group contains multiple second data entries. Then, the first group and its corresponding second group are compared. If the first group and the second group are inconsistent, the first data in the first group and the second data in the second group are compared.

[0080] The comparison between the first group in the first data copy and the second group in the second data copy is an overall comparison between the groups, which can only determine whether there is a difference between the first group and the second group, but cannot identify specific difference data. The comparison between each group is a comparison of the data in the group, which can identify the difference data.

[0081] The number of groups and the number of data entries in each group are determined by those skilled in the art based on the data size of the first and second data copies and the efficiency requirements for data consistency comparison.

[0082] The data correspondence refers to the correspondence between multiple data sets that need to be synchronized and whose values ​​need to be kept consistent. The mutual correspondence between groups means that there is a data correspondence between the first data in the first group and the second data in the second group.

[0083] It should be noted that the aforementioned first group does not refer to a specific group, but mainly refers to the various groups formed after the first data copy is grouped. The number of first groups is equal to the number of first groups in the first data copy. Similarly, the aforementioned second group does not refer to a specific group, but mainly refers to the various groups formed after the second data copy is grouped. The number of second groups is equal to the number of second groups in the second data copy.

[0084] The aforementioned first data does not refer to any specific data, but mainly refers to data derived from the first data copy. Similarly, the second data does not refer to any specific data, but mainly refers to data derived from the second data copy.

[0085] Based on the above preferred embodiments, this embodiment also provides a specific method for group comparison, namely, comparing the first group in the first data copy with the second group in the second data copy, specifically including:

[0086] Compare the group check codes of the first group with those of the second group to see if they match; wherein, based on generating data check codes for each data item in the corresponding group, the group check code of the group is generated according to the data check codes of all data in the group.

[0087] Specifically, generating the group check code of the group based on the data check codes of all data in the group involves: arranging the data check codes of the data in the group into a string in order, and generating the group check code based on the string. That is, after generating the data check code by performing the corresponding check code generation algorithm on the data itself, the corresponding check code generation algorithm is then performed on the data check code to generate the group check code.

[0088] The data check code or group check code is generated by one or more of the following algorithms: checksum, XOR check, CRC check, MD4, MD5, SHA-1, SHA-256, SHA-384, and SHA-512.

[0089] The above preferred embodiment optimizes the first comparison process to improve efficiency during the initial comparison. Building upon this, the embodiment further optimizes subsequent comparison processes to further improve efficiency. Specifically, it compares the first data copy with the corresponding first historical data copy to obtain a first modification record set, such as... Figure 3 As shown, it specifically includes:

[0090] In step 401, the third group in the first data copy is compared with the fourth group in the first historical data copy.

[0091] It should be noted that the third group has the same meaning as the first group mentioned above, except that the comparison objects here are different from those in step 301. For ease of understanding of the scheme, the third group is redefined here.

[0092] The aforementioned fourth group does not refer to a specific group, but mainly refers to the various groups formed after the first historical data copy is grouped. The first historical data copy is divided into several groups, and the first historical data copy contains several fourth groups.

[0093] In step 402, if the comparison shows that the third group and the fourth group are inconsistent, then the third data in the third group and the fourth data in the fourth group are compared one by one.

[0094] In step 403, if the comparison shows that the third data and the fourth data are inconsistent, a corresponding modification record is generated based on the third data and the fourth data and stored in the first modification record set.

[0095] The aforementioned third data does not refer to any specific data, but mainly refers to data derived from the first data copy. Similarly, the fourth data does not refer to any specific data, but mainly refers to data derived from the first historical data copy.

[0096] The third and fourth data are the same data at different times. The third data is the data value at the next comparison, and the fourth data is the data value at the previous comparison. During this period, the data value may change due to business calls. That is, if the third data and the fourth data are inconsistent, it is considered that the data has been modified during this period.

[0097] Specifically, generating the corresponding modification record based on the third data and the fourth data includes:

[0098] Using the fourth data as the data before modification and the third data as the data after modification, the data check code of the data before modification and the data check code of the data after modification are used as a modification record.

[0099] The comparison of the third group in the first data copy with the fourth group in the first historical data copy specifically includes:

[0100] The group check code of the third group is compared with the group check code of the fourth group to determine if they are consistent. The third group, which was used to divide the first data copy during the first comparison, is reused, and the group check code of the third group generated during the previous comparison is used as the group check code of the fourth group in the historical data copy for the next comparison.

[0101] At the same time, the data check code of the third data generated during the previous comparison will be used as the data check code of the fourth data during the next comparison.

[0102] Another preferred implementation method includes: if in each of the previous preset number of data consistency comparisons, multiple third groups are found to be consistent with the corresponding fourth group, the multiple third groups are merged, and the merged group is used for comparison in subsequent data consistency comparisons. The group merging can be done by directly merging multiple groups into one group and using the group checksum of that group for comparison, or by generating a total checksum based on the group checksums of multiple groups and comparing based on the total checksum. The preset number of comparisons, the number of groups to be merged, and the specific groups to be merged are determined by those skilled in the art based on factors such as the efficiency requirements of data comparison and the likelihood of data modification. For example, if multiple groups on the same page in the corresponding data table have been found to be unchanged in the past 8 consecutive data comparisons, the multiple groups on that page are merged, a page checksum is generated using the group checksum, and the page checksum is used for page comparison. If modification is found in the page comparison, the comparison proceeds to group comparison, and finally to data comparison to determine the modified data.

[0103] In an optional embodiment, data copies can be grouped in the following manner:

[0104] When comparing data on a table-by-table basis, the data segmentation size n is set, and the data page information P of table T in the data copy is obtained. The data page information P includes the table ID (TID), data page ID (PID), and file ID (FID). Then, table T is segmented in units of n, with each n data pages forming a group. Each group contains all the data from those n data pages, resulting in multiple groups. The group checksum of each group is calculated based on the checksums of all data within that group.

[0105] When the data in the first data copy and the second data copy are completely in one-to-one order, that is, when there is a data correspondence between the kth data in the first data copy and the kth data in the second data copy, the first data copy and the second data copy can be grouped using the same n value. When the data in the first data copy and the second data copy are not completely in one-to-one order, or when the correspondence order is misaligned, the data correspondence must be taken into consideration when grouping.

[0106] Based on the above embodiments, this embodiment also provides a preferred method for obtaining data comparison results, namely, obtaining the data comparison results between the first data copy and the second data copy based on historical comparison results, the first modification record set, and the second modification record set, such as... Figure 4 As shown, it specifically includes:

[0107] In step 501, the first modified record set and the second modified record set are deduplicated to remove duplicate modified records in the first modified record set and the second modified record set.

[0108] In step 502, the modified records in the first and second modified record sets after deduplication are taken as target modified records. The target modified records are compared with the historical comparison results to determine whether there are any comparison difference records in the historical comparison results that match the target modified records.

[0109] In this embodiment, the first and second modified record sets after deduplication can be integrated into a target modified record set. The target modified record set contains at least one target modified record, which belongs to either the first or the second modified record set. If the target modified record belongs to the first modified record set, it means that the first data copy has been modified while the second data copy has not been modified accordingly. If the target modified record belongs to the second modified record set, it means that the second data copy has been modified while the first data copy has not been modified accordingly.

[0110] In step 503, if the comparison difference record exists, it is deleted from the historical comparison results; if it does not exist, a corresponding comparison difference record is generated according to the target modification record, and the comparison difference record is added to the historical comparison results, so that the final historical comparison results are used as the data comparison results between the first data copy and the second data copy.

[0111] If there is a comparison difference record in the historical comparison results that matches the target modification record, it means that the first data copy and the second data copy have performed the same modification operation successively. Although there was a difference between the two in the previous comparison, they have both performed the same modification operation in this comparison, and the comparison difference record needs to be deleted from the historical comparison results.

[0112] The identical modification record specifically refers to the following: the first data copy and the second data copy both modify the first data and the second data that have a data correspondence relationship, and the value of the first data before modification is consistent with the value of the second data before modification, and the value of the first data after modification is consistent with the value of the second data before modification. At this time, the modification of the first data and the modification of the second data are synchronized, and it is considered that there is no difference in their consistency.

[0113] Specifically, determining whether there is a comparison difference record in the historical comparison results that matches the target modification record includes:

[0114] When the target modified record belongs to the first modified record set, it is determined whether the first data in the comparison difference record is consistent with the data before modification of the target modified record, and whether the second data in the comparison difference record is consistent with the data after modification of the target modified record. If both are consistent, then there is a comparison difference record that matches the target modified record in the historical comparison results.

[0115] In this case, it means that the first data has been modified to be synchronized with the second data. Specifically, it only checks whether the comparison difference records that correspond to the first and second data match.

[0116] When the target modified record belongs to the second modified record set, it is determined whether the second data in the comparison difference record is consistent with the data before modification of the target modified record, and whether the first data in the comparison difference record is consistent with the data after modification of the target modified record. If both are consistent, then there is a comparison difference record that matches the target modified record in the historical comparison results.

[0117] In this case, it means that the second data has been modified to be synchronized with the first data. Specifically, only records showing a correlation with the first and second data are checked for matching differences.

[0118] The step involves generating a corresponding comparison difference record based on the target modification record, specifically including:

[0119] When the target modified record belongs to the first modified record set, the modified data of the target modified record is used as the first data, the corresponding second data is searched, and a comparison difference record is generated.

[0120] When the target modified record belongs to the second modified record set, the modified data of the target modified record is used as the second data, the corresponding first data is searched, and a comparison difference record is generated.

[0121] The comparison difference record may directly store the first data and the second data, or it may store the data check code of the first data and the data check code of the second data.

[0122] Specifically, the search for the corresponding second data includes:

[0123] In the historical comparison results, it is checked whether there is a historical comparison difference record corresponding to the first data. If there is, a comparison difference record is generated using the first data and the second data in the historical comparison difference record. If there is no, a comparison difference record is generated using the data before the modification of the target modification record as the second data.

[0124] If there are no historical comparison differences, it means that the first data was synchronized with the second data before the modification, and their values ​​were the same. Therefore, when generating comparison difference records, the data is directly used instead of searching for it separately.

[0125] Based on the same concept, the search for the corresponding first data specifically includes:

[0126] In the historical comparison results, it is checked whether there is a historical comparison difference record corresponding to the second data. If there is, a comparison difference record is generated using the second data and the first data in the historical comparison difference record; if there is no, a comparison difference record is generated using the data before the modification of the target modification record as the second data.

[0127] In this embodiment, "first," "second," and "third" do not have any special meaning. They are used only to facilitate the description of different individuals within a class of objects, and should not be interpreted as having any special meaning in terms of order or other aspects.

[0128] Example 2:

[0129] Based on the method described in Embodiment 1, this invention combines specific application scenarios and uses technical descriptions in relevant scenarios to illustrate the implementation process of the features of this invention in those scenarios.

[0130] like Figure 5The illustration shows an application scenario of the present invention, in which replica B is a complete copy of the data in replica A. After the copying is completed, replica A is accessed and modified by service A, and replica B is accessed and modified by service B. The two services are independent of each other and may modify the data in their respective replicas at any time. Replica A can be understood as the first data replica of Embodiment 1, and replica B can be understood as the second data replica of Embodiment 1.

[0131] In this scenario, one data consistency comparison method is as follows: Figure 6 As shown, it specifically includes:

[0132] In step 601, during the first comparison, copies A and B are compared to obtain the MD5 value storage file FILE for copy A, the MD5 value storage file FILE for copy B, and a data difference report REP. The REP contains the MD5 hash (DMD5A) present in copy A but not in copy B, and the MD5 hash (DMD5B) present in copy B but not in copy A. When both DMD5A and DMD5B are empty, it indicates that the data in copies A and B are consistent; otherwise, it indicates that there is a data difference. The DMD5A and DMD5B data are saved to the local disk. The MD5 value storage files FILE for copies A and B can be understood together as the storage files used in Example 1 for storing data checksums and group checksums, for subsequent comparisons.

[0133] In step 602, during subsequent comparisons, the FILE files of copy A and copy B generated in the previous comparison are read, and the current copy A and copy B are compared with their corresponding FILE files to obtain the corresponding modified data (which can be understood as the first modification record set and the second modification record set). Based on the modified data and the previous data difference report REP (which can be understood as the historical comparison result), the final comparison difference report is generated.

[0134] Among them, step 601 above is as follows Figure 7 As shown, it specifically includes:

[0135] In step 701, data comparison is performed on a table-by-table basis. Before data comparison, the data segmentation size n needs to be set. First, the data page information P of table T in the copy is obtained. P includes table id (TID), data page id (PID), and file id (FID). Then, P is segmented in units of n to obtain a data page grouping set G{G0…Gm}, where G0 is the set {P0…Pn} and G1 is the set {Pn+1…P2n}. Each element in the data page grouping set G contains a unique number. G0 represents element number 0, G1 represents element number 1, and Gm represents element number m. The elements in the grouping set G can be understood as the groups in Example 1. One group contains all the data corresponding to multiple data pages, such as... Figure 8 As shown.

[0136] In step 702, each row of data in the database has a globally unique Rowid. The Rowid information accurately locates the position of the data row. The Rowid is an 18-byte string containing the file ID (FID), table ID (TID), page ID (PID), and row ID within the page (RID). The Rowid generation function FUN_ROWID is used to generate the corresponding Rowid for each data record based on the information (FID, TID, PID, RID), as follows: Figure 9 As shown.

[0137] In step 703, the range of rowid is calculated for each element in the data page grouping set G. Due to database characteristics, a data page can store a maximum of 65535 rows of data, so the range of id values ​​within a data page is [0, 65535]. The range of rowid in element Gi{Pni, Pni+1, ..., P(n+1)i-1} is [rowid of the 0th row of the first data page of Gi, rowid of the 0th row of the first data page of Gi+1], such as... Figure 10 As shown.

[0138] In step 704, a corresponding SQL statement is generated for each element Gi. Since each Gi only contains a portion of the data in the table, filtering conditions need to be added to the SQL to filter the data. During the data reading process, the table data may have changed, causing the data page information P obtained in step 1 to change. Therefore, when reading the data, an open range method needs to be used to read all the data in the table, such as... Figure 11 As shown. Therefore, the SQL generated by the first element G0[Rowidmin0, Rowidmax0] is as follows:

[0139] SELECT*FROM T WHERE ROWID<=Rowidmax0;

[0140] The SQL generated by the last element Gm[Rowidminm, Rowidmaxm] is as follows:

[0141] SELECT*FROM T WHERE ROWID>=Rowidminm;

[0142] The SQL generated by other elements Gi[Rowidmini, Rowidmaxi] is as follows: SELECT * FROM T WHERE ROWID >= Rowidmini and ROWID <Rowidmin(i+1)。

[0143] In step 705, a data reading thread is created to read the data contained in element Gi. For each row of data, an MD5 value is calculated, resulting in a set of row data MD5 values ​​iMD5{iMD50, iMD51…iMD5n} (which can be understood as the set of data checksums in Example 1). An MD5 value is then calculated for all MD5 values ​​in the GMD5 set, resulting in the GMD5 (GMD5i) of element Gi (which can be understood as the group checksum in Example 1). The first data page information, IMD5, and GMD5 value of elements Gi and Gi+1 are saved to the local file FILEi. The Gi+1 information of the last element is set to empty. The filename includes the element number information of Gi to ensure the uniqueness of the file. Figure 12 As shown.

[0144] In step 706, the FILEi file of copy A is compared with the FILEi file of copy B to obtain a data difference report REP (which can be understood as a data comparison result). The REP contains the md5 hash (DMD5A) that exists in copy A but not in copy B (which can be understood as the data checksum of the first data), and the md5 hash (DMD5B) that exists in copy B but not in copy A (which can be understood as the data checksum of the second data). When both DMD5A and DMD5B are empty, it indicates that the data in copy A and copy B are consistent; otherwise, it indicates that there is a data difference. The DMD5A and DMD5B data are then saved to the local disk, as follows: Figure 13 As shown.

[0145] The above step 602 is as follows Figure 14 As shown, it specifically includes:

[0146] In step 801, the FILEi file generated from the previous comparison is read to obtain the Gi information and GMD5 (GMD5_OLDi) from the previous comparison. Based on the Gi page information in the FILEi file, an SQL statement is generated, such as... Figure 15 As shown.

[0147] In step 802, MD5 comparison threads CMP_DEC_MD5_THR and CMP_INC_MD5_THR are created respectively. A data reading thread is created to read the data contained in element Gi, calculate the MD5 value for each row of data, and obtain the set of row data MD5 values ​​iMD5_NEW{iMD50, iMD51…iMD5n}. MD5 values ​​are calculated for all MD5 values ​​in GMD5 to obtain the GMD5(GMD5_NEWi) of element Gi. GMD5_NEWi (which can be understood as the group check code of the corresponding data copy in Example 1) is compared with GMD5_OLDi (which can be understood as the group check code of the corresponding historical data copy in Example 1). If they are the same, it means that the data in element Gi has not changed and does not need to be sent to the comparison thread for comparison; if GMD5_NEWi and GMD5_OLDi are different, it means that there is a data change in Gi; proceed to step 803.

[0148] In step 803, the iMD5 information (iMD5_OLD) in FILEi is obtained, and the difference between iMD5_OLD (which can be understood as the data checksum of the corresponding historical data copy in Embodiment 1) and iMD5_NEW (which can be understood as the data checksum of the corresponding data copy in Embodiment 1) is compared. Data DIF_DEC, which exists in iMD5_OLD but not in iMD5_NEW, is obtained; data DIF_INC, which exists in iMD5_NEW but not in iMD5_OLD, is obtained; DIF_DEC is sent to the comparison thread CMP_DEC_MD5_THR, and DIF_INC is sent to CMP_INC_MD5_THR; the FILEi file of Gi is updated. Here, DIF_DEC and DIF_INC can be collectively understood as the first modification record or the second modification record in Embodiment 1, where DIF_DEC represents the data before modification and DIF_INC represents the data after modification.

[0149] In step 804, the CMP_DEC_MD5_THR comparison thread collects DIF_DEC data and compares whether the DIF_DEC of replica A and replica B are the same. If they are the same, it means that the data in replica A and replica B has decreased by the same amount compared to the last comparison. If they are different, it means that the data in replica A and replica B has decreased by different amounts compared to the last comparison. At this time, the excess DIF_DEC data in replica A is added to DDMD5A, and the excess DIF_DEC data in replica B is added to DDMD5B.

[0150] In step 805, the CMP_INC_MD5_THR comparison thread collects DIF_INC data and compares whether the DIF_INC of copy A and copy B are the same. If they are the same, it means that the data in copy A and copy B has increased by the same amount of data compared to the last comparison. If they are different, it means that the data in copy A and copy B has increased by different amounts of data compared to the last comparison. At this time, the DIF_INC data that copy A has more data is added to DIMD5A, and the DIF_INC data that copy B has more data is added to DIMD5B. DDMD5A, DDMD5B, DIMD5A, and DIMD5B can be collectively understood as the set of target modification records in embodiment 1, or the organized set of target modification records. DDMD5A represents the set of data before modification corresponding to the first data copy, DDMD5B represents the set of data after modification corresponding to the first data copy, DIMD5A represents the set of data after modification corresponding to the first data copy, and DIMD5B represents the set of data after modification corresponding to the second data copy.

[0151] In step 806, the previous data comparison result difference reports DMD5A and DMD5B are obtained (DMD5A and DMD5B can be collectively understood as the historical comparison results in Example 1). It is checked whether the data in DMD5A exists in DMD5A. If it exists, it indicates that the inconsistent data from the previous comparison has been deleted from copy A during this comparison. If the data in DMD5A is not in DMD5A, it indicates that copy A deleted additional data during this comparison, and this data needs to be added to DMD5B. Figure 16 As shown; similarly, check if the data in DDMD5B exists in DMD5B. If it does not exist, add the data to DMDBA.

[0152] In step 807, check if the data in DIMD5A exists in DMD5B. If it exists, it indicates that the data is newly generated difference data in this comparison. This data already existed in copy B during the last comparison, so delete the identical DM5 values ​​in DIMD5A and DMD5B. If the data in DIMD5A does not exist in DMD5B, it indicates that the data is newly added difference data, so add it to DMD5A. Figure 17 As shown; similarly, delete the same MD5 in DIMD5B and DMD5A, add the redundant MD5 value in DIMD5B to DMD5B; update the local disk files of DMD5A and DMD5B (which can be understood as the final data comparison result in Example 1).

[0153] Example 3:

[0154] The application scenarios of the methods described in Embodiments 1 and 2 are mainly in the data consistency comparison between two dynamically modified data copies. Based on the same concept as Embodiments 1 and 2, this invention proposes a data consistency comparison method between dynamically modified data copies and static data copies. This method is applicable to application scenarios where multiple business data copies are derived from a source data copy, and the source data copy is not modified due to business operations, but is only modified through synchronization with each derived business data copy.

[0155] The method specifically includes:

[0156] The business data copy is compared with the corresponding historical business data copy to obtain the business modification record set.

[0157] Based on the business modification record set and the historical comparison results, the data comparison results between the business data copy and the source data copy are obtained.

[0158] Specifically, the business data copy used in the previous comparison is used as the historical business data copy for the next comparison; during the first comparison, the business data copy is compared with the source data copy to obtain the data comparison result; and when synchronizing the source data copy, the corresponding data comparison result is updated to ensure consistency with the synchronized source data copy.

[0159] The specific implementation of the above methods is based on the same concept as in Example 1, and will not be repeated here.

[0160] Example 4:

[0161] The methods described in Examples 1-3 are primarily used for data consistency comparison between two data replicas. Based on the same concept as the methods described in Examples 1-3, this invention proposes a method for data consistency comparison between multiple data replicas, specifically including:

[0162] The business data copy is compared with the corresponding historical business data copy to obtain the corresponding historical business modification records.

[0163] The data comparison results are obtained based on the historical business modification records and historical comparison results of multiple business data copies.

[0164] Specifically, the business data copy used in the previous comparison is used as the historical business data copy for the next comparison, and the data comparison result obtained in the previous comparison is used as the historical comparison result for the next comparison. During the first comparison, based on the data correspondence among multiple business data copies, the data in all business data copies with data correspondence are compared. If the data values ​​in at least one copy are different, it is considered that there is a difference in data consistency, and all data values ​​with data correspondence are stored in the data comparison result.

[0165] The specific implementation of the comparison process is based on the same concept as in Example 1, but differs from Example 1 in that:

[0166] The modified record is removed only when the same modified record exists in the modification record set of all business data copies, so that it does not participate in the subsequent data comparison result generation process.

[0167] When determining whether there is a comparison difference record that matches the corresponding modification record in the historical comparison results, the corresponding data values ​​of each data copy in the comparison difference record are compared to confirm whether the modified value in the modification record set of each data copy is consistent with the value of other data copies that have not been modified in the difference record. When the data values ​​of all data copies are consistent, it is determined to be a match.

[0168] Example 5:

[0169] like Figure 18 The diagram shown is a schematic representation of the architecture of a data consistency comparison device according to an embodiment of the present invention. The data consistency comparison device of this embodiment includes one or more processors 21 and a memory 22. Figure 18 Take a processor 21 as an example.

[0170] Processor 21 and memory 22 can be connected via a bus or other means. Figure 18 Taking the example of a connection between China and Israel via a bus.

[0171] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the data consistency comparison method in Embodiment 1. The processor 21 executes the data consistency comparison method by running the non-volatile software program and instructions stored in the memory 22.

[0172] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may optionally include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0173] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, they perform the data consistency comparison method described in Embodiment 1 above, for example, performing the above-described... Figures 1-4 , Figure 6 , Figure 7 and Figure 14 The steps shown.

[0174] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0175] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0176] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data consistency comparison method, characterized in that, include: The first data copy is compared with the corresponding first historical data copy to obtain the first set of modification records; The second data copy is compared with the corresponding second historical data copy to obtain the second set of modification records; Based on the historical comparison results, the first modification record set, and the second modification record set, the data comparison results between the first data copy and the second data copy are obtained; In this process, the data copy used in the previous comparison is used as the historical data copy for the next comparison, and the data comparison result obtained in the previous comparison is used as the historical comparison result for the next comparison. In the first comparison, the first data copy is compared with the second data copy to obtain the data comparison result. The step of obtaining the data comparison result between the first data copy and the second data copy based on the historical comparison results, the first modification record set, and the second modification record set specifically includes: The first and second modified record sets are deduplicated to remove duplicate modified records from both sets. The modified records in the first and second modified record sets after deduplication are used as target modified records. The target modified records are compared with historical comparison results to determine if there are any comparison difference records in the historical comparison results that match the target modified record. Specifically, when the target modified record belongs to the first modified record set, it is determined whether the first data in the comparison difference record is consistent with the data before modification of the target modified record, and whether the second data in the comparison difference record is consistent with the data after modification of the target modified record. If both are consistent, then there is a comparison difference record matching the target modified record in the historical comparison results. When the target modified record belongs to the second modified record set, it is determined whether the second data in the comparison difference record is consistent with the data before modification of the target modified record, and whether the first data in the comparison difference record is consistent with the data after modification of the target modified record. If both are consistent, then there is a comparison difference record matching the target modified record in the historical comparison results. If it exists, the comparison difference record is deleted from the historical comparison results; if it does not exist, a corresponding comparison difference record is generated according to the target modification record, and the comparison difference record is added to the historical comparison results, so that the final historical comparison results are used as the data comparison results between the first data copy and the second data copy.

2. The data consistency comparison method according to claim 1, characterized in that, The step of comparing the first data copy with the second data copy to obtain the data comparison result specifically includes: Compare the first packet in the first data copy with the second packet in the second data copy; If the comparison shows that the first group and the second group are inconsistent, the first data in the first group and the second data in the second group are compared one by one to determine the first and second data that are inconsistent with each other. In this process, the data in the data copy is divided into multiple groups, each group containing multiple data entries. When dividing the data into groups, the data correspondence between different data copies is used to make the groups obtained from different data copies correspond to each other. Based on the inconsistencies between the first and second data, corresponding comparison difference records are generated and stored in the data comparison results.

3. The data consistency comparison method according to claim 2, characterized in that, The comparison of the first packet in the first data copy with the second packet in the second data copy specifically includes: Compare the group check codes of the first group with those of the second group to see if they match; wherein, based on generating data check codes for each data item in the corresponding group, the group check code of the group is generated according to the data check codes of all data in the group.

4. The data consistency comparison method according to claim 3, characterized in that, The step of comparing the first data copy with the corresponding first historical data copy to obtain the first modification record set specifically includes: Compare the third group in the first data copy with the fourth group in the first historical data copy; If the comparison shows that the third group and the fourth group are inconsistent, then the third data in the third group and the fourth data in the fourth group are compared one by one. If the comparison shows that the third data and the fourth data are inconsistent, then a corresponding modification record is generated based on the third data and the fourth data and stored in the first modification record set.

5. The data consistency comparison method according to claim 4, characterized in that, The step of generating corresponding modification records based on the third data and the fourth data specifically includes: Using the fourth data as the data before modification and the third data as the data after modification, the data check code of the data before modification and the data check code of the data after modification are used as a modification record.

6. The data consistency comparison method according to claim 4, characterized in that, The comparison of the third group in the first data copy with the fourth group in the first historical data copy specifically includes: The group check code of the third group is compared with the group check code of the fourth group to determine if they are consistent. The third group, which was used to divide the first data copy during the first comparison, is reused, and the group check code of the third group generated during the previous comparison is used as the group check code of the fourth group in the historical data copy for the next comparison.

7. The data consistency comparison method according to claim 1, characterized in that, The step involves generating a corresponding comparison difference record based on the target modification record, specifically including: When the target modified record belongs to the first modified record set, the modified data of the target modified record is used as the first data, the corresponding second data is searched, and a comparison difference record is generated. When the target modified record belongs to the second modified record set, the modified data of the target modified record is used as the second data, the corresponding first data is searched, and a comparison difference record is generated.

8. A data consistency comparison device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the data consistency comparison method according to any one of claims 1-7.

Images