A data comparison method and device based on large table data segmentation

Abstract

The present invention relates to a data comparison method and device based on large table data segmentation. The method mainly includes: dividing the data of two tables to be compared into the same number of parts; applying for a complete binary tree with the same number of leaf nodes as the number of divided parts, and each leaf node records a pair of data to be compared; comparing the data of each leaf node to obtain inconsistent data, and comparing the inconsistent data pairwise until only the last part of inconsistent data remains. The method adopted by the present invention can greatly save the comparison time and improve the comparison efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of database data processing, and particularly to a data comparison method and device based on large table data segmentation. Background Art

[0002] With the rapid development of informatization construction, the business is becoming more and more complex, and the coupling between various businesses is also becoming stronger. Often, different businesses need to access the same data. In this case, the data accessed by each business often becomes hot data, and the competition for data access will lead to a decline in business performance. To solve this problem, the usual processing method is to make multiple copies of this data, and each business accesses different data copies. In this case, the data consistency of each copy is particularly important.

[0003] Data comparison is usually carried out in units of tables, and the comparison method includes the following three steps:

[0004] 1. Read the data of the comparison table from the copies to be compared respectively.

[0005] 2. Perform MD5 encryption calculation on the read data in units of data rows.

[0006] 3. Compare the MD5s of the two sides of the data to find inconsistent MD5 values.

[0007] In step 1, usually a single thread is used to read the data of the comparison table. When the data volume is small, the time taken to read the data of the comparison table is short. However, when the data volume of a single table is huge, the following problems exist:

[0008] 1. The time taken to read the data of a single table is too long, affecting the efficiency of data comparison.

[0009] 2. In a multi-version database system, some data in the query result set needs to be obtained from the UNDO (undo) space. When the data reading time is too long, some data in the UNDO space will be overwritten by the database system, resulting in a reading failure. At the same time, step 3 can only be carried out after all the data has been obtained, affecting the data comparison efficiency.

[0010] In view of this, how to overcome the defects of the existing technology and solve the above technical problems is a difficult problem to be solved in this technical field. Summary of the Invention

[0011] In view of the above deficiencies or improvement requirements of the prior art, the present invention provides a data comparison method and device based on large table data segmentation. For a table with a large amount of data, the data is segmented into multiple small data sets. When acquiring data, a complete binary tree with the same number of leaf nodes as the number of segments is applied to simultaneously acquire different data sets, thereby accelerating the data acquisition speed. At the same time, the comparison is performed in the manner of the leaf nodes of the binary tree. Each leaf node is equivalent to a thread to compare a group of data sets, and the inconsistent data enters the next round of comparison, which can greatly save the comparison time and improve the comparison efficiency.

[0012] The embodiments of the present invention adopt the following technical solutions:

[0013] In a first aspect, the present invention provides a data comparison method based on large table data segmentation, including:

[0014] Both of the two table data to be compared are segmented into the same number of parts;

[0015] Apply a complete binary tree with the same number of leaf nodes as the number of segmentation parts, and each leaf node records a pair of data to be compared;

[0016] Compare the data of each leaf node to obtain inconsistent data, and compare the inconsistent data pairwise until only the last piece of inconsistent data remains.

[0017] Further, the step of segmenting both of the two table data to be compared into the same number of parts specifically includes:

[0018] Obtain the number m of result sets for data segmentation from the configuration;

[0019] Query the table number, the file number of the data page occupied by data storage, and the data page number of the table to be compared from the database copy, and segment the data pages into m parts in the order of the file number and the data page number.

[0020] Further, the specific process of segmenting the data pages into m parts includes:

[0021] Obtain the number n of data pages of the table;

[0022] Determine that the number of data pages of each segmented data is t = n / m;

[0023] Data segmentation step: For each segmented data, let n = n - t, m = m - 1;

[0024] Judge whether m = 0. If so, end the segmentation process; if not, return to the data segmentation step.

[0025] Further, after splitting the data page into m parts, calculate the starting ROWID of each part of the data according to the starting page in each part of the data, and generate a query statement for each part of the data based on the starting ROWID of this part of the data and the starting ROWID of the next part of the data, so as to obtain the result set of this part of the data through this query statement.

[0026] Further, apply for a complete binary tree with the same number of leaf nodes as the number of split parts, and each leaf node records a pair of data to be compared, specifically including:

[0027] After splitting the data of the two tables to be compared into m parts, apply for a complete binary tree with m leaf nodes, and record the i-th pair of data subsets to be compared in the i-th leaf node; where i is an integer greater than or equal to 1 and less than or equal to m.

[0028] Further, comparing the data of each leaf node to obtain inconsistent data, and pairwise comparing the inconsistent data until only the last piece of inconsistent data remains, specifically including:

[0029] Compare the data of each initial leaf node to obtain the inconsistent data of the first copy of the table data and the inconsistent data of the second copy of the table data, and use the two inconsistent data compared by each node as a group of differential data;

[0030] Input every two groups of differential data into a new leaf node for comparison to obtain a new group of differential data, and repeat this step until the comparison of the last leaf node is completed, leaving only the last group of differential data, that is, the last piece of inconsistent data;

[0031] Output this last piece of inconsistent data to generate a comparison difference report.

[0032] Further, the structure of the leaf node includes Result1->Db1_left, Result1->Db2_left, Result2->Db1_left, and Result2->Db2_left, where Result1 represents the data of the left node of the binary tree, Result2 represents the data of the right node of the binary tree, Db1_left represents the inconsistent data in the first copy of the table data after comparison, and Db2_left represents the inconsistent data in the second copy of the table data after comparison.

[0033] Further, in the initial state, the Result1->Db1_left of the i-th leaf node records the i-th data subset read from the first data copy of the table to be compared, the Result1->Db2_left of the i-th leaf node is empty, the Result2->Db1_left of the i-th leaf node is empty, and the Result2->Db2_left of the i-th leaf node records the i-th data subset read from the second data copy of the table to be compared.

[0034] Further, when each leaf node performs comparison, Result1->Db1_left is compared with Result2->Db2_left, and Result1->Db2_left is compared with Result2->Db1_left. The two Db1_lefts obtained from the comparison are merged to form a new Db1_left, and the two Db2_lefts obtained from the comparison are merged to form a new Db2_left. The merged Db1_left and Db2_left are used as a new set of differential data and input into a new leaf node for the next round of comparison.

[0035] On the other hand, the present invention provides a data comparison device based on large table data segmentation, specifically including: at least one processor and a memory, the at least one processor and the memory are connected through a data bus, and the memory stores instructions executable by the at least one processor. After the instructions are executed by the processor, they are used to complete the data comparison method based on large table data segmentation in the first aspect.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows: for a table with a large amount of data, the data is segmented into multiple small data sets. When acquiring data, a complete binary tree with the same number of leaf nodes as the number of segments is applied to simultaneously acquire different data sets, thereby accelerating the data acquisition speed. At the same time, the comparison is performed in the manner of the leaf nodes of the binary tree. Each leaf node is equivalent to a thread to compare a set of data sets therein. The inconsistent data enters the next round of comparison, which can greatly save the comparison time and improve the comparison efficiency. Description of the Drawings

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly introduced below. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0038] Figure 1 It is a flowchart of a data comparison method based on large table data segmentation provided in Embodiment 1 of the present invention;

[0039] Figure 2 The specific flowchart of step 100 provided by Embodiment 1 of the present invention;

[0040] Figure 3 The specific flowchart of dividing a data page into m parts provided by Embodiment 1 of the present invention;

[0041] Figure 4 The exemplary diagram of determining the ROWID range provided by Embodiment 1 of the present invention;

[0042] Figure 5 The specific flowchart of step 300 provided by Embodiment 1 of the present invention;

[0043] Figure 6 The schematic diagram of the specific memory structure of the leaf nodes of the binary tree provided by Embodiment 1 of the present invention;

[0044] Figure 7 The schematic diagram of the leaf nodes obtaining data provided by Embodiment 1 of the present invention;

[0045] Figure 8 The specific comparison schematic diagram of the leaf nodes provided by Embodiment 1 of the present invention;

[0046] Figure 9 The schematic diagram of the specific comparison process of each leaf node provided by Embodiment 1 of the present invention;

[0047] Figure 10 The schematic diagram of the specific comparison process between Db1 and Db2 provided by Embodiment 2 of the present invention;

[0048] Figure 11 The schematic diagram of the structure of a data comparison device based on large table data segmentation provided by Embodiment 3 of the present invention. Specific implementation manners

[0049] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] The present invention is an architecture of a specific function system. Therefore, in specific embodiments, the functional logic relationships of each structural module are mainly described, and the specific software and hardware implementation manners are not limited.

[0051] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will be described in detail below with reference to the drawings and embodiments.

[0052] Embodiment 1:

[0053] For the convenience of description, in the embodiments of the present invention, the transactions in the source - end logs are referred to as "single transactions", and the transactions formed after merging multiple single transactions are referred to as "multiple transactions".

[0054] As Figure 1 shown, the embodiments of the present invention provide a data comparison method based on large - table data segmentation, and the specific steps are as follows.

[0055] Step 100: Divide the data of the two tables to be compared into the same number of parts. This step is for tables with a large amount of data, and the data is divided into multiple small data sets. In this embodiment, after determining the number of small data sets for division, the number of data sets of the two copies to which the data of the two tables to be compared belong is the same after the table data is divided.

[0056] Step 200: Apply for a complete binary tree with the number of leaf nodes equal to the number of divided parts, and each leaf node records a pair of data to be compared. In this step, when obtaining data, by applying for a complete binary tree with the number of leaf nodes equal to the number of divisions, different data sets can be obtained simultaneously, which can speed up the data acquisition speed.

[0057] Step 300: Compare the data of each leaf node to obtain inconsistent data, and compare the inconsistent data pairwise until only the last piece of inconsistent data remains. This step uses the leaf - node method of the binary tree for comparison. Each leaf node is equivalent to a thread to compare a set of data sets, and the inconsistent data enters the next round of comparison, which can greatly save the comparison time and improve the comparison efficiency.

[0058] Through the above steps, in this preferred embodiment, the original large - table data is divided into small data sets one by one, and then the corresponding small data sets are obtained simultaneously through the leaf nodes of the binary tree, and the small data sets are compared through the leaf nodes. Such a comparison method can greatly save the comparison time and improve the comparison efficiency compared with the traditional comparison method in the background technology.

[0059] Specifically, as Figure 2 shown, in this preferred embodiment, for the above - mentioned step 100 (divide the data of the two tables to be compared into the same number of parts), this preferred embodiment can expand it into the following steps:

[0060] Step 101: Obtain the number m of the result set for data segmentation from the configuration. The number m of the result set obtained from the configuration in this step is also the number of small data sets designed in advance to be divided.

[0061] Step 102: Query the table number of the table to be compared, the file number of the data page occupied by data storage, and the data page number from the database replicas. Divide the data pages into m parts in the order of the file number and the data page number. This step can operate on the table data of two database replicas simultaneously to save time. For example, if the table data to be compared are the table data of replica Db1 and replica Db2 respectively, then the table data of replica Db1 and replica Db2 can be processed simultaneously. Query the table number TABID of the table to be compared, the file number FNO of the data page occupied by data storage, and the data page number BLOCK_ID, and divide the data pages into m parts in the order of FNO and BLOCK_ID.

[0062] For the specific process of dividing the data pages into m parts, as Figure 3 shown, the specific process is as follows. After obtaining the division number m, obtain the number of data pages n of the table; determine that the number of data pages of each divided data is t = n / m (that is, the number of data pages of the i-th divided data is t = n / m, where i is an integer between 1 and m); then enter the data division step: for each divided data, let n = n - t and m = m - 1; determine whether m = 0. If so, it means that the data pages have been divided and the division process ends. If not, return to the data division step, continue to divide one data, and then let n = n - t and m = m - 1 and re-determine whether m = 0. Keep looping until m = 0 and then end the division process.

[0063] Based on the above process, in this preferred embodiment, after dividing the data pages into m parts, calculate the starting ROWID of each part of the data according to the starting page in each part of the data, and generate a query statement for each part of the data according to the starting ROWID of this part of the data and the starting ROWID of the next part of the data, so as to obtain the result set of this part of the data through this query statement. The function for calculating ROWID according to TABID, FNO, and BLOCK_ID in this process is called FRID.

[0064] As Figure 4As shown in the figure, it is an example diagram for calculating the starting ROWID and determining the ROWID range in this preferred embodiment. In this example, the number of partitions m = 3, that is, the table T1 is divided into three small data sets, and then the starting ROWID of each small data set is calculated through the FRID function respectively. For example, the starting ROWID of the first small data set in the figure is: AAAVo7AAEAAAAITAAA, the starting ROWID of the second small data set is: AAAVo7AAEAAAAIVAAA, and the starting ROWID of the third small data set is: AAAVo7AAEAAAAIXAAA. Then, the ROWID interval range of the first small data set can be obtained as: [AAAVo7AAEAAAAITAAA, AAAVo7AAEAAAAIVAAA), the ROWID interval range of the second small data set is: [AAAVo7AAEAAAAIVAAA, AAAVo7AAEAAAAIXAAA), and the ROWID interval range of the third small data set is: [AAAVo7AAEAAAAIXAAA, +∞). It should be noted that since there is no starting ROWID of the next data set for the last small data set, +∞ is used instead. After obtaining the ROWID interval of each small data set, the query statement for this copy of data can be generated according to the ROWID interval range, so as to obtain the result set of this copy of data through this query statement, and the obtained result set is also the data of this small data set.

[0065] In this preferred embodiment, for the above step 200 (applying for a complete binary tree with the number of leaf nodes equal to the number of partitions, and each leaf node records a pair of data to be compared), this preferred embodiment can be embodied as: after dividing the data of the two tables to be compared into m parts, applying for a complete binary tree with the number of leaf nodes being m, and recording the i-th pair of data subsets to be compared in the i-th leaf node; where i is an integer greater than or equal to 1 and less than or equal to m. In this embodiment, when the data of mpart[i] is obtained, mpart[i] is incremented by one. When mpart[i] = 2, it means that the i-th subsets of the replica DB1 and the replica DB2 have both been obtained and can be compared.

[0066] As Figure 5 shown, in this preferred embodiment, step 300 (comparing the data of each leaf node to obtain inconsistent data, and pairwise comparing the inconsistent data until only the last piece of inconsistent data remains) can be specifically expanded into the following steps:

[0067] Step 301: Compare the data of each initial leaf node to obtain the inconsistent data of the first copy of the table data and the inconsistent data of the second copy of the table data, and take the two inconsistent data compared by each node as a group of difference data.

[0068] Step 302: Input every two sets of differential data into a new leaf node for comparison to obtain a new set of differential data. Repeat this step until the comparison of the last leaf node is completed, leaving only the last set of differential data, i.e., the last inconsistent data. The comparison result of the last node in this step is the comparison result of the entire comparison task.

[0069] Step 303: Output the last inconsistent data to generate a comparison difference report.

[0070] As Figure 6 shown, it is a schematic diagram of the specific memory structure of the leaf nodes of the binary tree in this preferred embodiment. Among them, the structure of the leaf node includes Result1->Db1_left, Result1->Db2_left, Result2->Db1_left, and Result2->Db2_left. Among them, Db1_left represents the inconsistent data in the first table data copy after comparison, Db2_left represents the inconsistent data in the second table data copy after comparison, Result1 represents the data of the left node of the binary tree, Result2 represents the data of the right node of the binary tree, and flag represents the data filling situation. flag = 2 indicates that the data of Result1 and Result2 has been filled.

[0071] In the initial state, the Result1->Db1_left of the i-th leaf node records the i-th data subset read from the first data copy of the table to be compared, the Result1->Db2_left of the i-th leaf node is empty, the Result2->Db1_left of the i-th leaf node is empty, and the Result2->Db2_left of the i-th leaf node records the i-th data subset read from the second data copy of the table to be compared.

[0072] As Figure 7 shown, it is a schematic diagram of the leaf node obtaining data. The structure of the leaf node in the figure includes Result1->Db1_left, Result1->Db2_left, Result2->Db1_left, and Result2->Db2_left. When obtaining data, both the left and right nodes of the leaf node use a query statement within the ROWID range to obtain data sets from the copies Db1 and Db2 respectively. After successful acquisition, because the leaf node is in the initial state at this time, the data recorded by Result1->Db1_left is the data obtained from the copy Db1, while Result1->Db2_left is empty. Similarly, the data recorded by Result2->Db2_left is the data obtained from the copy Db2, while Result2->Db1_left is empty.

[0073] The following uses a specific example of leaf node comparison to further illustrate step 300 (steps 301 - 303). As Figure 8 shown, assume that the number of divided data sets m = 5, then the initial number of leaf nodes is also 5. Figure 8 Among them, Node1, Node2, Node3, Node4, and Node5 respectively represent the initial 5 leaf nodes. The structure of each initial leaf node is Result1->Db1_left, NULL, NULL, Result2->Db2_left. The first round of comparison is the comparison of the initial 5 leaf nodes. After comparing Result1->Db1_left and Result2->Db2_left of each leaf node, a set of inconsistent data is formed, that is, the inconsistent data of replica Db1 and the inconsistent data of replica Db2. Then, a set of inconsistent data compared from the first leaf node is recorded in the left node of the new leaf node (Node6) as Result1->Db1_left and Result1->Db2_left in Node6, and a set of inconsistent data compared from the second leaf node is recorded in the right node of the new leaf node (Node6) as Result2->Db1_left and Result2->Db2_left in Node6. Similarly, the inconsistent data after comparing the 3rd and 4th leaf nodes is recorded in the new leaf node (Node7) for the second round of comparison. It should be noted that when the number of leaf nodes in the same round is odd, there will be a set of inconsistent data compared in the next round that is skipped. When encountering skipped data, it can be delayed by one more round for comparison. For example, in this example, for the 5th leaf node, since the 1st - 4th nodes are just paired in twos, the inconsistent data of the 5th node is skipped, and the new nodes Node6 and Node7 formed after comparing the 1st - 4th nodes are also just paired in twos, so the 5th leaf node is skipped again. When comparing Node8 formed after comparing Node6 and Node7, at this time, only one more set of inconsistent data can be generated, and the inconsistent data of the 5th leaf node (Node5) and the inconsistent data of Node8 can be recorded in a new leaf node (Node9) for comparison. Finally, the inconsistent data compared by this Node9 is also the finally compared difference data, and generating a comparison difference report for this difference data can complete the data comparison.

[0074] In the above comparison process, the specific comparison process for each leaf node is as Figure 9 shown: From Figure 9It can be seen that Db1_left and Db2_left in each Result of the node are the results of comparison in the previous node. Therefore, when comparing each leaf node, Result1->Db1_left only needs to be compared with Result2->Db2_left, and Result2->Db1_left only needs to be compared with Result1->Db2_left. In the initial leaf node, both Result2->Db1_left and Result1->Db2_left are empty, and the comparison results are the same. In the comparison of leaf nodes in other rounds, a set of redundant data in Db1 and a set of redundant data in Db2 will be compared between Result1->Db1_left and Result2->Db2_left, and a set of redundant data in Db1 and a set of redundant data in Db2 will also be compared between Result2->Db1_left and Result1->Db2_left. At this time, the two Db1_left (the redundant data in Db1) obtained by comparison need to be merged to form a new Db1_left, and the two Db2_left (the redundant data in Db2) obtained by comparison need to be merged to form a new Db2_left. The merged Db1_left and Db2_left are used as a new set of differential data and input into the left or right node of the new leaf node for the next round of comparison.

[0075] In summary, in this embodiment, for a table with a large amount of data, the data is divided into multiple small data sets. When obtaining data, a complete binary tree with the same number of leaf nodes as the number of divisions is applied to obtain different data sets simultaneously, which speeds up the data acquisition speed. At the same time, the comparison is carried out in the way of the leaf nodes of the binary tree. Each leaf node is equivalent to a thread to compare a set of data sets among them. The inconsistent data enters the next round of comparison, which can greatly save the comparison time and improve the comparison efficiency.

[0076] Embodiment 2:

[0077] Based on the data comparison method based on large table data segmentation provided in Embodiment 1, this Embodiment 2 further illustrates the present invention through a specific implementation scenario.

[0078] In this embodiment, the table T1 (tabid = 88635) of the replica Db1 contains 5 data pages (531 - 535), and all the data pages are in the file with file number fno = 4. The specific data distribution is as follows:

[0079] Page number Data range 531 1～100 532 101～200 533 201～300 534 301～400 535 401～500

[0080] The copy of table T1 in Db2 (tabid = 88699) contains 6 data pages (575 - 580), and all the data pages are in the file with file number fno = 4. The specific data distribution is as follows:

[0081] Page number Data range 575 0～99 576 100～199 577 200～300 578 301～400 579 401～500 580 501-600

[0082] Assume m = 3, then in DB1, the number of data pages for each partition is [1, 2, 2]. The results after partitioning are as follows:

[0083] Data segmentation Page number Data range Part1 531 1～100 Part2 532～533 101～300 Part3 534～535 301～500

[0084] In DB2, the number of data pages for each partition is [2, 2, 2]. The results after partitioning are as follows:

[0085] Data segmentation Page number Data range Part1 575-576 0～199 Part2 577-578 200～400 Part3 579-580 401～600

[0086] The specific comparison process is as Figure 10 shown. First, create 3 initial leaf nodes Node1, Node2, and Node3. In Node1, Result1->Db1_left, Result1->Db2_left, Result2->Db1_left, and Result2->Db2_left record the data 1 - 100, NULL, NULL, 0 - 199 in sequence; in Node2, Result1->Db1_left, Result1->Db2_left, Result2->Db1_left, and Result2->Db2_left record the data 101 - 300, NULL, NULL, 200 - 400 in sequence; in Node3, Result1->Db1_left, Result1->Db2_left, Result2->Db1_left, and Result2->Db2_left record the data 301 - 500, NULL, NULL, 401 - 600 in sequence.

[0087] Compare the data in Node1, Node2, and Node3. Among them, the results of the inconsistent data compared in Node1 can be listed as NULL (Db1 inconsistent data), NULL (Db2 inconsistent data), NULL (Db1 inconsistent data), 0, 101 - 199 (Db2 inconsistent data) in sequence. Merge the Db1 inconsistent data and the Db2 inconsistent data. Finally, the inconsistent data compared in Node1 is NULL (Db1 inconsistent data), 0, 101 - 199 (Db2 inconsistent data).

[0088] For Node2, the results of the inconsistent data identified by the comparison can be listed in sequence as 101 - 199 (Db1 inconsistent data), NULL (Db2 inconsistent data), NULL (Db1 inconsistent data), 301 - 400 (Db2 inconsistent data). After merging the Db1 inconsistent data and the Db2 inconsistent data, the inconsistent data identified by Node2 is finally 101 - 199 (Db1 inconsistent data), 301 - 400 (Db2 inconsistent data).

[0089] For Node3, the results of the inconsistent data identified by the comparison can be listed in sequence as 301 - 400 (Db1 inconsistent data), NULL (Db2 inconsistent data), NULL (Db1 inconsistent data), 501 - 600 (Db2 inconsistent data). After merging the Db1 inconsistent data and the Db2 inconsistent data, the inconsistent data identified by Node3 is finally 301 - 400 (Db1 inconsistent data), 501 - 600 (Db2 inconsistent data).

[0090] After the first round of comparison is completed, the second round of comparison begins. The inconsistent data identified by Node1 and Node2 is recorded in a new leaf node (Node4). At this time, the inconsistent data identified by Node3 is bypassed and not recorded in the new leaf node for the time being. For Node4 in the second round of comparison, the data recorded in its Result1->Db1_left, Result1->Db2_left, Result2->Db1_left, and Result2->Db2_left are NULL, 0, 101 - 199, 101 - 199, and 301 - 400 in sequence. After comparing them, the results of the inconsistent data identified by the comparison can be listed in sequence as NULL (Db1 inconsistent data), 0 (Db2 inconsistent data), NULL (Db1 inconsistent data), 301 - 400 (Db2 inconsistent data). After merging the Db1 inconsistent data and the Db2 inconsistent data, the inconsistent data identified by Node4 is finally NULL (Db1 inconsistent data), 0, 301 - 400 (Db2 inconsistent data).

[0091] After the second round of comparison is completed, the third round of comparison begins. At this time, the inconsistent data compared by Node4 can be paired with the inconsistent data compared by Node3 that was left out in the previous round. The data of both is recorded in a new leaf node (Node5). After recording, the values of Result1->Db1_left, Result1->Db2_left, Result2->Db1_left, and Result2->Db2_left of Node5 are recorded as NULL, 0, 301 - 400, 301 - 400, and 501 - 600 in sequence. Comparing Node5, the results of the inconsistent data obtained can be listed as NULL (Db1 inconsistent data), 0 (Db2 inconsistent data), NULL (Db1 inconsistent data), 501 - 600 (Db2 inconsistent data) in sequence. The Db1 inconsistent data is merged, and the Db2 inconsistent data is merged. Finally, the inconsistent data compared by Node5 is NULL (Db1 inconsistent data), 0, 501 - 600 (Db2 inconsistent data). This inconsistent data is output as a comparison report to complete all comparisons.

[0092] In summary, in this embodiment, the data is divided into multiple small data sets. When acquiring data, a complete binary tree with the same number of leaf nodes as the number of divisions is applied to acquire different data sets simultaneously, accelerating the data acquisition speed. At the same time, the comparison is carried out in the way of the leaf nodes of the binary tree. Each leaf node is equivalent to a thread to compare a set of data sets, and the inconsistent data enters the next round of comparison, which can greatly save the comparison time and improve the comparison efficiency.

[0093] Embodiment 3:

[0094] Based on the data comparison method based on large table data division provided in the above Embodiment 1 to Embodiment 2, the present invention further provides a data comparison device based on large table data division that can be used to implement the above method. As Figure 11 shown, it is a schematic diagram of the device architecture of an embodiment of the present invention. The data comparison device based on large table data division in this embodiment includes one or more processors 21 and a memory 22. Among them, Figure 11 one processor 21 is taken as an example.

[0095] The processor 21 and the memory 22 can be connected through a bus or other means. Figure 11 Taking the connection through the bus as an example.

[0096] The memory 22 serves as a non-volatile computer-readable storage medium and can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the data comparison method and system based on large table data segmentation in Embodiments 1 to 2. The processor 21 executes various functional applications and data processing of the data comparison device based on large table data segmentation by running the non-volatile software programs, instructions, and modules stored in the memory 22, that is, implements the data comparison method based on large table data segmentation in Embodiments 1 to 2.

[0097] The memory 22 may include high-speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 optionally includes a memory remotely disposed relative to the processor 21, and these remote memories can be connected to the processor 21 through a network. Examples of the above network include but are not limited to the Internet, intranet, local area network, mobile communication network, and combinations thereof.

[0098] The program instructions / modules are stored in the memory 22 and, when executed by one or more processors 21, execute the data comparison method based on large table data segmentation in Embodiments 1 to 2 above. For example, execute each of the Figure 1 steps shown above.

[0099] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the embodiments can be completed by instructing relevant hardware through a program, and this program can be stored in a computer-readable storage medium. The storage medium may include: Read Only Memory (ROM), Random Access Memory (RAM), magnetic disk, optical disk, etc.

[0100] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data comparison method based on large table data segmentation, characterized in that, Including: Dividing both of the two table data to be compared into the same number of parts; Applying for a full binary tree with the number of leaf nodes equal to the number of divided parts, and recording a pair of data to be compared in each leaf node; Comparing the data of each leaf node to obtain inconsistent data, and pairwise comparing the inconsistent data until only the last piece of inconsistent data remains; specifically, comparing the data of each initial leaf node to obtain the inconsistent data of the first table data copy and the inconsistent data of the second table data copy, and taking the two pieces of inconsistent data compared from each node as a set of differential data; inputting every two sets of differential data into a new leaf node for comparison to obtain a new set of differential data, repeating this step until the comparison of the last leaf node is completed and only the last set of differential data, that is, the last piece of inconsistent data, remains; outputting the last piece of inconsistent data to generate a comparison difference report; The structure of the leaf node includes Result1->Db1_left, Result1->Db2_left, Result2->Db1_left, and Result2->Db2_left, where Result1 represents the data of the left node of the binary tree, Result2 represents the data of the right node of the binary tree, Db1_left represents the inconsistent data in the first table data copy after comparison, and Db2_left represents the inconsistent data in the second table data copy after comparison; in the initial state, Result1->Db1_left of the i-th leaf node records the i-th data subset read from the first table data copy to be compared, Result1->Db2_left of the i-th leaf node is empty, Result2->Db1_left of the i-th leaf node is empty, and Result2->Db2_left of the i-th leaf node records the i-th data subset read from the second table data copy to be compared; when each leaf node conducts a comparison, Result1->Db1_left is compared with Result2->Db2_left, and Result1->Db2_left is compared with Result2->Db1_left. The two compared Db1_left are merged to form a new Db1_left, and the two compared Db2_left are merged to form a new Db2_left. The merged Db1_left and Db2_left are used as a new set of differential data and input into a new leaf node for the next round of comparison.

2. The data comparison method based on large table data segmentation according to claim 1, wherein The specific process of dividing both of the two table data to be compared into the same number of parts includes: Obtaining the number m of result sets for data division from the configuration; Querying the table number, the file number of the data page occupied by data storage, and the data page number of the table to be compared from the database copy, and dividing the data pages into m parts in the order of the file number and the data page number.

3. The data comparison method based on large table data segmentation according to claim 2, wherein The specific process of dividing the data pages into m parts includes: Obtaining the number n of data pages of the table; Determining that the number of data pages of each divided data is t = n / m; Data splitting step: For each split of data, let n = n - t and m = m - 1; Judge whether m = 0. If so, end the splitting process; otherwise, return to the data splitting step.

4. The data comparison method based on large table data segmentation according to claim 2, characterized in that After splitting the data pages into m parts, calculate the starting ROWID of each part of the data according to the starting page in each part of the data, and generate a query statement for each part of the data based on the starting ROWID of this part of the data and the starting ROWID of the next part of the data, so as to obtain the result set of this part of the data through this query statement.

5. The data comparison method based on large table data segmentation according to claim 2, characterized in that, Apply for a complete binary tree with the same number of leaf nodes as the number of splitting parts, and each leaf node records a pair of data to be compared, specifically including: After splitting the two table data to be compared into m parts, apply for a complete binary tree with m leaf nodes, and record the i-th pair of data subsets to be compared in the i-th leaf node; where i is an integer greater than or equal to 1 and less than or equal to m.

6. A data comparison device based on large table data splitting, characterized in that: It includes at least one processor and a memory, the at least one processor and the memory are connected through a data bus, the memory stores instructions executable by the at least one processor, and after the instructions are executed by the processor, they are used to complete the data comparison method based on large table data splitting according to any one of claims 1-5.

Images