A database table comparison method and system that can reduce lock usage

Abstract

The application discloses a database table comparison method and device capable of reducing lock use, and the method comprises the following steps: setting a local Hash chain table for a source and a target respectively, and respectively hashing the exported data into the local Hash chain tables; setting two global Hash tables corresponding to the source and the target for each database table respectively, and inserting a memory block in the local Hash chain table into the corresponding chain table of the global Hash table of the local Hash chain table when the memory block is full; generating a comparison request task when the total memory block number of a pair of chain tables of the global Hash table exceeds a limit block; taking out the sub-chain table corresponding to the Hash value in the comparison request task from the global Hash table of the source and the target respectively when the comparison thread receives the comparison request task, and performing deduplication calculation; and submitting a final comparison request task for each pair of sub-chain tables of the global Hash tables of the source and the target after the whole export task is completed.

Description

Technical Field

[0001] The present invention relates to the technical field of database synchronization, and in particular to a database table comparison method and system capable of reducing lock usage. Background Art

[0002] Database software is an indispensable component of computer software. To meet requirements such as database disaster recovery and read-write separation, database synchronization software is often deployed.

[0003] When synchronizing data with database software, it's inevitable that the source and target databases may have inconsistent data content or data quantity. To detect inconsistencies in a timely manner, data comparison is necessary.

[0004] The strategy used by database synchronization software when comparing tables determines performance. This strategy is particularly important for large tables. A well-designed strategy can consistently control computational overhead and memory usage while maintaining high comparison efficiency.

[0005] In the prior art, database synchronization software commonly uses the following strategy when comparing tables: A common strategy involves using hash linked lists to eliminate duplicates. This strategy involves hashing the MD5 values ​​of each data record in the database table and assigning the records to different linked lists. When a record is to be inserted into the hash table, the linked list is first determined using the hash. The target linked list is then checked to see if there is a duplicate MD5 in the other linked list (the target linked list). If so, the record is removed from both parties. If not, the record is inserted into the target linked list.

[0006] This processing strategy requires locking the target linked list when each row is inserted into the hash table to protect the consistency of the data in memory and avoid access conflicts. This leads to frequent lock use and a high probability of lock conflicts, which is particularly evident when the database table is large. In addition, this practice of allocating memory for a single record seriously affects performance and is prone to memory conflicts. Summary of the Invention

[0007] To overcome the deficiencies of the above-mentioned prior art, the purpose of the present invention is to provide a database table comparison method and system that can reduce the use of locks, and to improve the speed of database table comparison by optimizing the use of locks by database synchronization software during the database table comparison process.

[0008] To achieve the above object, the present invention provides a database table comparison method that can reduce lock usage, comprising the following steps:

[0009] Step S1: Setting a first local Hash list for each of the source and target export threads. The export threads of both parties hash the exported data into their respective first local Hash lists according to a hash method.

[0010] Step S2: setting two global hash tables for each database table, one corresponding to the source end and the other corresponding to the target end;

[0011] Step S3: Two sub-linked lists with the same hash value in the global hash table of the source end and the target end are regarded as a pair of linked lists. When the total number of memory blocks of a pair of sub-linked lists in the global hash table of the source end and the target end exceeds the limit block, the export thread generates a comparison request task, requesting the comparison thread to compare the corresponding sub-linked lists, and the comparison request task includes the hash value of the corresponding sub-linked list;

[0012] Step S4: When the comparison thread receives the comparison request task, it retrieves the source and target sub-linked tables corresponding to the hash values ​​in the comparison request task from the global hash tables of the source and target ends, performs a concatenation operation on each data record in the sub-linked tables, and performs deduplication calculation;

[0013] Step S5: After each deduplication calculation is completed, the comparison thread again attempts to retrieve the sub-list with the same hash value as the one in task S4 from the global hash table. If new data is obtained, the deduplication calculation continues. If no new data is obtained, the remaining data is placed back into the global hash tables of the source and target ends.

[0014] Step S6: When the residual data of the source and target ends in the comparison thread exceeds a set value, write the data to the file corresponding to the hash value in the S4 task;

[0015] Step S7: After the entire export task is completed, a final comparison request task is submitted for each pair of sub-linked tables of the global hash table on the source and target ends, and the comparison thread performs a final comparison on the data written to the cache disk file.

[0016] Optionally, in step S1, each export thread hashes the md5 value of the exported data record and hashes it to the first local Hash linked list at its respective end according to the hash result; when a memory block in a first local Hash linked list is full, the memory block is inserted into the corresponding linked list of the corresponding global Hash table.

[0017] Optionally, the first local hash linked list is composed of several head pointers, each head pointer points to the corresponding memory block block to form a sub-linked list, each sub-linked list has a head pointer pointing to the corresponding memory block, each memory block has a next pointer and several records, each record contains the md5 value of the record and the row_ind structure, the row_ind structure locates the specific position of a record, which is reserved for locating the record during comparison.

[0018] Optionally, in step S1 , the first 10 bits of the first two bytes of the 16-byte md5 value of the derived data record are used as the conditional hash value to be hashed into the corresponding local hash table.

[0019] Optionally, if the corresponding Hash linked list has not allocated a memory block, a memory block is allocated and the record is stored; if a memory block has been allocated, the record is directly stored in the corresponding memory block.

[0020] Optionally, in step S2, the head pointer of the global Hash linked list points to the first memory block, and each memory block contains a next pointer and several data records, and the next pointer points to the next memory block.

[0021] Optionally, in step S1, when a certain memory block in the first local Hash linked list is full, the corresponding global Hash table is locked in the process of inserting the memory block into the corresponding linked list of the global Hash table on one side.

[0022] Optionally, in step S4, when the comparison thread receives the comparison request task, it takes out the source and target terminal linked lists corresponding to the hash values ​​in the comparison request task from the global hash linked lists of the source and target ends and puts them into the comparison local hash table, and concatenates each data record according to the hash value using the row_ind structure.

[0023] Optionally, in step S4, the first 10 bits of the last two bytes of each data record are used as a hash value, and the data records are connected in series using a row_ind structure according to the hash value.

[0024] To achieve the object, the present invention further provides a database table comparison system capable of reducing lock usage, comprising:

[0025] The export thread includes a source-side export thread and a target-side export thread, which are used to set a first local Hash linked list for each source-side export thread and a target-side export thread respectively, and hash the exported data into their respective first local Hash linked lists according to the hash method. The export thread also sets two global Hash tables for each database table, corresponding to the source end and the target end respectively. During the data export process, when a memory block in the local Hash linked list is full, the memory block is inserted into the corresponding linked list of the global Hash table on its side. When the total number of memory blocks in a pair of linked lists in the global Hash table exceeds the limit block, a comparison request task is generated, requesting the comparison thread to compare the corresponding linked lists. After the entire export task is completed, a final comparison request task is submitted for each pair of sub-linked lists of the global Hash table of the source end and the target end;

[0026] The comparison thread is used to, when receiving a comparison request task, take out the source and target sub-linked lists corresponding to the Hash value in the comparison request task from the global Hash table of the source and target ends respectively, connect each data record in the sub-linked list in series according to the Hash value using the row_ind structure, and perform deduplication calculation. When each deduplication calculation is completed, the comparison thread tries again to get the sub-linked list with the same Hash value as the comparison request task from the global Hash table. If new data is obtained, the deduplication calculation continues. If there is no new data, the remaining data is put back into the global Hash table of the source and target ends; when the calculated residual memory block exceeds a specific value, the data is written to the disk file corresponding to the Hash value.

[0027] Compared with the prior art, the database table comparison method and system of the present invention can reduce the use of locks, significantly reducing the frequency of lock usage during the database table comparison process. Locks are only used during access operations on the global hash table and the allocation and release of memory blocks. The lock usage is reduced to one-256th of the original amount. The present invention improves the speed of database table comparison by optimizing the use of locks by database synchronization software during the database table comparison process. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of the steps of a database table comparison method for reducing lock usage according to the present invention;

[0029] Figure 2 This is a memory diagram of the first local Hash linked list in a specific embodiment of the present invention;

[0030] Figure 3 This is a memory diagram of a global Hash linked list in a specific embodiment of the present invention;

[0031] Figure 4This is a memory diagram of a comparison local hash table within a comparison thread in a specific embodiment of the present invention;

[0032] Figure 5 This is a system architecture diagram of a database table comparison system that can reduce lock usage according to the present invention. DETAILED DESCRIPTION

[0033] The following describes the embodiments of the present invention using specific examples and accompanying drawings. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through other different specific examples, and the details in this specification may be modified and altered based on different viewpoints and applications without departing from the spirit of the present invention.

[0034] Figure 1 This is a flowchart of the steps of a database table comparison method that can reduce lock usage in the present invention. Figure 1 As shown, the present invention provides a database table comparison method that can reduce lock usage, comprising the following steps:

[0035] Step S1: Set a first local Hash linked list for the source and target export threads respectively. The export threads of both parties hash the exported data into their respective first local Hash linked lists according to the Hash method.

[0036] In this invention, data export and comparison on the source and target sides are separated into an export thread and a comparison thread. The export threads on both sides hash the exported data into their respective first local hash lists using a hashing method. Specifically, the export threads hash the MD5 values ​​of the exported data records and, based on the hash results, hash them into their respective first local hash lists.

[0037] Specifically, in order to reduce memory allocation conflicts and reduce the use of memory allocation locks, the present invention no longer allocates separate memory for each record, but allocates a memory block for every N (for example, 256) records. The memory block is referred to as a block. Figure 2 As shown, the first local Hash linked list consists of several head pointers, each of which points to a corresponding memory block to form a sub-linked list. That is, in each sub-linked list, there is a head pointer pointing to a corresponding memory block. Each memory block contains a next pointer and 0-256 records. Each record contains the md5 value of the corresponding record and a row_ind structure. The row_ind structure is as follows:

[0038]

[0039] The row_ind structure locates the specific position of a record and is reserved for locating records during comparison. In the row_ind structure, blk represents the pointer to the memory block of the record, and idx represents the number of the record in the memory block.

[0040] In a specific embodiment of the present invention, the specific approach is: some bits in the 16-byte md5 value of the exported data record are used as the conditional hash value to the corresponding first local hash table (that is, the source end and the target end each have their own first local hash table, and the exported data records are hashed into their own first local hash tables respectively). The hash range of this process is called HASH_L1. In some embodiments, the first 10 bits of the first two bytes of the 16-byte md5 value of the exported data record are used as the hash value by default, that is, the default range is 0-1023. For example, if the first 10 bits of the first two bytes of the md5 value of a data record are 1, the data record should be stored in the sub-linked list corresponding to 1. If no memory block is allocated on the sub-linked list of the corresponding first local hash linked list, a memory block block is allocated and the data record is stored. If a memory block block has been allocated, the data record is directly stored in the corresponding memory block.

[0041] In this invention, the first local hash list is used only within the thread for internal data caching. It does not perform matching calculations or require locking. Because separate memory is no longer allocated for each record, instead a memory block is allocated for every N (e.g., 256) records. Therefore, the memory allocation lock usage in this invention is reduced to one-tenth of the original N (e.g., 256).

[0042] Step S2: Set up two global hash tables for each database table, one corresponding to the source end and the other corresponding to the target end. When a memory block in the first local hash table is full, the memory block is inserted into the end of the corresponding sub-list of the global hash table on the other side according to the hash value.

[0043] In a specific embodiment of the present invention, the memory structure of the global Hash linked list is as follows: Figure 3As shown, the head pointer of the global hash list points to the first memory block. Each memory block contains a next pointer and 0-256 records. The next pointer points to the next memory block, forming a sub-linked list. When a memory block in a first local hash list is full, the memory block is inserted into the tail of the corresponding sub-linked list with the same hash value in the local global hash table. For example, if a memory block in the first local hash list on the source side is full, the memory block is inserted into the global hash table on the source side according to the hash value. This process requires locking the corresponding global hash table to ensure data consistency and avoid access conflicts. This shows that the locking frequency of the present invention is one-tenth of the previous N (e.g., 256).

[0044] Step S3, treats the two sub-linked lists with the same hash value in the global hash table of the source end and the target end as a pair of linked lists. When the total number of memory blocks of a pair of sub-linked lists in the global hash table of the source end and the target end exceeds the limit block, the export thread generates a comparison request task, requesting the comparison thread to compare the corresponding sub-linked lists. The comparison request task contains the hash value of the corresponding sub-linked list.

[0045] Step S4: When the comparison thread receives the comparison request task, it takes out the source and target terminal sub-linked lists corresponding to the hash values ​​in the comparison request task from the global hash linked lists of the source and target ends respectively, connects each data record in the sub-linked list in series according to the hash value using the row_ind structure, and performs deduplication calculation.

[0046] In a specific embodiment of the present invention, two local Hash tables are set up inside the comparison thread, corresponding to the source and target ends respectively, to complete the comparison of specific comparison tasks. When the comparison thread receives a comparison request task, it takes out the source and target end linked tables corresponding to the Hash value in the comparison request task from the global Hash table of the source and target ends (and deletes them from the global Hash table at the same time), and concatenates each record using the row_ind structure according to the different Hash value from the previous one. Figure 4As shown, for example, if the Hash value of a row of records in the sub-linked table taken out from the source end is 1, then first determine whether the row_ind structure at the position where the Hash value is 1 in the source end comparison local Hash table is empty. If it is empty, the blk pointer in the structure points to the memory block idx and records the specific position of the record in the memory block. If it is not empty, find the corresponding record according to the row_ind structure, use the row_ind structure in the corresponding record to mark the position of the record, repeat in sequence, concatenate the records, and perform deduplication calculations. Therefore, the present invention no longer involves the problem of memory allocation. The Hash range of the internal comparison local Hash table is HASH_L2, and the default range is 0-1023 (in this embodiment, the first 10 bits of the last two bytes are used as the Hash value by default). The purpose of this is that the Hash values ​​of the two sub-linked tables taken out this time are the same. In order to be able to hash into the comparison local Hash table for comparison, a new Hash range is used.

[0047] In step S5, after each deduplication calculation is completed, the comparison thread again attempts to retrieve the sublist with the same hash value as in task S4 from the global hash tables on the source and target ends. If new data is retrieved, the deduplication calculation continues, repeating step S4. If no new data is found, the remaining data is placed back into the corresponding global hash tables on the source and target ends, that is, the remaining data is inserted into the end of the sublist with the same hash value (based on HASH_L1) in the global hash table. Because the data on the source and target ends may differ, there may be residual data that cannot be deduplicated, and this residual data needs to be placed back into the corresponding global hash tables.

[0048] Step S6: When the residual memory blocks calculated by the source and target ends in the comparison thread exceed a specific value (set value), the residual data are written to the file corresponding to the hash value in the S4 task.

[0049] For example, if the number of memory blocks in the comparison thread exceeds 100, the remaining data will be written to the file corresponding to the hash value in the S4 task. That is to say, when the remaining memory blocks exceed a specific value, a file corresponding to the hash value in the S4 task will be created. This is done to prevent too much data in the memory. When the memory occupancy is too large, writing this data into a file can reduce the data in the memory, thereby reducing the memory occupancy.

[0050] In step S7, after the entire export task is completed, the export thread submits a final comparison request task for each pair of sublists in the global hash table on the source and target ends. This task compares each pair of sublists in memory with the data written to the cache disk file. The data in memory is compared first, followed by the data in the disk file.

[0051] That is, after the entire export task is completed, a final comparison task is submitted for each pair of source and target linked lists in the global hash table. After the comparison thread compares the in-memory data, it performs its final cleanup work, beginning with the comparison of the cached disk file. When the disk file is retrieved and sent to the comparison thread, due to the large data volume, the hash range is expanded to shorten the length of the sublists in the comparison thread's local hash table. The hash range used is HASH_L3, with a default range of 0-512KB (the first 19 bits of the first three bytes are used as the hash value by default).

[0052] Figure 5 This is a system architecture diagram of a database table comparison system that can reduce lock usage in the present invention. Figure 5 As shown, the present invention provides a database table comparison system capable of reducing lock usage, comprising:

[0053] The export thread 51 includes a source-side export thread and a target-side export thread, which are used to set up local Hash linked lists for the source-side and target-side export threads respectively, and hash the exported data into their respective local Hash linked lists according to the Hash method. The export thread 51 also sets up two global Hash tables for each database table, corresponding to the source and target ends respectively. During the data export process, when a memory block in the local Hash linked list is full, the memory block is inserted into the corresponding linked list of the global Hash table on its side. When the total number of memory blocks in a pair of linked lists in the global Hash table exceeds the limit block, a comparison request task is generated, requesting the comparison thread to compare the corresponding linked lists. After the entire export task is completed, a final comparison request task is submitted for each pair of sub-linked lists of the source-side and target-side global Hash tables, that is, the data written to the cache disk file is finally compared.

[0054] In this invention, data export and comparison on the source and target sides are separated into an export thread and a comparison thread. The export threads on both sides then hash the exported data into their respective local hash lists. Specifically, the source / target export threads hash the MD5 values ​​of the exported data records and, based on the hash results, hash them into their respective local hash lists.

[0055] Specifically, in order to reduce memory allocation conflicts and the use of memory allocation locks, the present invention no longer allocates separate memory for each record, but instead allocates a memory block for every N (for example, 256) records. This memory block is referred to as a block. The local hash linked list consists of several head pointers, each of which points to a corresponding memory block to form a sub-linked list. That is, each sub-linked list contains a head pointer pointing to a corresponding memory block. Each memory block contains a next pointer and 0-256 records. Each record contains the md5 value of the record and a row_ind structure. The row_ind structure locates the specific position of a record and is reserved for locating records during comparison. In the row_ind structure, blk represents a pointer to the memory block of the record, and idx represents the number of the record in the memory block. In a specific embodiment of the present invention, the specific approach is: some bits in the 16-byte md5 value of the exported data record are used as conditional hash values ​​to the corresponding local hash table. The hash range of this process is called HASH_L1, and the default range is 0-1023 (the first 10 bits of the first two bytes of the 16-byte md5 value of the exported data record are used as the hash value by default). If the corresponding hash linked list has not allocated a memory block, a memory block block is allocated and the record is stored. If a memory block block has been allocated, the record is directly stored in the corresponding memory block.

[0056] In this invention, the local hash list is used only within the thread for internal data caching. No matching calculations are performed, and no locking is required. Because separate memory is no longer allocated for each record, instead a memory block is allocated for every N (e.g., 256) records. Therefore, the memory allocation lock usage in this invention is reduced to one-tenth of the original N (e.g., 256).

[0057] In a specific embodiment of the present invention, the memory structure of the global Hash linked list is as follows: Figure 3 As shown, the head pointer of the global hash list points to the first memory block. Each memory block contains a next pointer and 0-256 records. The next pointer points to the next memory block. Each memory block forms a sub-linked list. When a memory block in a local hash list is full and inserted into the corresponding linked list of the local global hash table, the corresponding global hash table needs to be locked to ensure data consistency and avoid access conflicts. This shows that the locking frequency of the present invention is one-tenth of the previous N (e.g., 256).

[0058] When the export thread finds that the total number of memory blocks of a pair of linked lists in the global hash table exceeds the limit, it generates a comparison request task and requests the comparison thread to compare the corresponding linked lists.

[0059] The comparison thread 52 is used to, when receiving a comparison request task, take out the source and target sub-linked lists corresponding to the Hash value in the comparison request task from the global Hash linked lists of the source and target ends respectively, concatenate each data record in the sub-linked list using the row_ind structure according to the Hash value, and perform deduplication calculation. When each deduplication calculation is completed, the comparison thread 52 will try again to obtain the sub-linked list with the same Hash value as the comparison request task. If new data is obtained, the deduplication calculation will continue. If there is no new data, the remaining data will be put back into the global Hash table of the source and target ends; when the calculated residual memory block exceeds a specific value, the data will be written to the disk file corresponding to the Hash value.

[0060] In a specific embodiment of the present invention, a comparison local Hash table is set up inside the comparison thread 52 to complete the comparison of a specific comparison task. When the comparison thread 52 receives the comparison request task from the export thread, it takes out the source and target terminal linked tables corresponding to the specific Hash value from the global Hash linked tables of the source and target ends and puts them into the comparison local Hash table, and concatenates each record according to the Hash value using the row_ind structure, as shown in the following example: Figure 4 As shown, deduplication calculation is performed, so the present invention no longer involves memory allocation issues. The Hash range of the internal comparison local Hash table is HASH_L2, and the default range is 0-1023 (in this embodiment, the first 10 bits of the last two bytes are used as the Hash value by default).

[0061] After each deduplication calculation is completed, the comparison thread tries again to retrieve the data of the specific linked list from the global hash table. If new data is obtained, the deduplication calculation continues. If there is no new data, the remaining data is put back into the global hash table.

[0062] When the calculated remaining memory block exceeds a specific value, the data is written to the file corresponding to the hash value.

[0063] When the final comparison request task is received, the comparison thread performs a final comparison on the data written to the cache disk file.

[0064] It can be seen that the present invention significantly reduces the frequency of lock usage during database table comparison. Locks are only used in the allocation and release of memory blocks during access operations in the global hash table, and the lock usage is 1 / 256 of the original.

[0065] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Any skilled artisan may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be as set forth in the appended claims.

Claims

1. A database table comparison method that can reduce lock usage includes the following steps: Step S1: Setting a first local Hash list for each of the source and target export threads. The export threads of both parties hash the exported data into their respective first local Hash lists according to a hash method. Step S2: setting two global hash tables for each database table, one corresponding to the source end and the other corresponding to the target end; Step S3: Two sub-linked lists with the same hash value in the global hash table of the source end and the target end are regarded as a pair of linked lists. When the total number of memory blocks of a pair of sub-linked lists in the global hash table of the source end and the target end exceeds the limit block, the export thread generates a comparison request task, requesting the comparison thread to compare the corresponding sub-linked lists, and the comparison request task includes the hash value of the corresponding sub-linked list; Step S4: When the comparison thread receives the comparison request task, it retrieves the source and target sub-linked tables corresponding to the hash values ​​in the comparison request task from the global hash tables of the source and target ends, performs a concatenation operation on each data record in the sub-linked tables, and performs deduplication calculation; Step S5: After each deduplication calculation is completed, the comparison thread again attempts to retrieve the sub-list with the same hash value as the one in task S4 from the global hash table. If new data is obtained, the deduplication calculation continues. If no new data is obtained, the remaining data is placed back into the global hash tables of the source and target ends. Step S6: When the residual data of the source and target ends in the comparison thread exceeds a set value, write the data to the file corresponding to the hash value in the S4 task; Step S7: After the entire export task is completed, a final comparison request task is submitted for each pair of sub-lists in the global hash table on the source and target ends, and the comparison thread performs a final comparison on the data written to the cache disk file; In step S1, when a certain memory block in the first local Hash linked list is full, the memory block is inserted into the corresponding linked list of the global Hash table on one side, and the corresponding global Hash table is locked.

2. A database table comparison method capable of reducing lock usage according to claim 1, characterized in that: In step S1, each export thread hashes the md5 value of the exported data record and hashes it to the first local Hash linked list of each end according to the hash result; when a memory block in a first local Hash linked list is full, the memory block is inserted into the corresponding linked list of the corresponding global Hash table.

3. A database table comparison method capable of reducing lock usage according to claim 2, characterized in that: The first local hash linked list is composed of several head pointers, each head pointer points to the corresponding memory block block to form a sub-linked list, each sub-linked list has a head pointer pointing to the corresponding memory block, each memory block has a next pointer and several records, each record contains the md5 value of the record and the row_ind structure, the row_ind structure locates the specific position of a record, which is reserved for locating the record during comparison.

4. A database table comparison method capable of reducing lock usage according to claim 3, characterized in that: In step S1 , the first 10 bits of the first two bytes of the 16-byte md5 value of the derived data record are used as the hash value to conditionally hash into the corresponding local hash table.

5. A database table comparison method capable of reducing lock usage according to claim 4, characterized in that: If the corresponding Hash linked list has not allocated a memory block, allocate a memory block and store the record. If the memory block has been allocated, store the record directly in the corresponding memory block.

6. A database table comparison method capable of reducing lock usage according to claim 4, characterized in that: In step S2, the head pointer of the global hash table points to the first memory block. Each memory block contains a next pointer and several data records, and the next pointer points to the next memory block.

7. A database table comparison method capable of reducing lock usage according to claim 6, characterized in that: In step S4, when the comparison thread receives the comparison request task, it takes out the source and target terminal linked lists corresponding to the hash values ​​in the comparison request task from the global hash tables of the source and target ends and puts them into the comparison local hash table, and concatenates each data record according to the hash value using the row_ind structure.

8. A database table comparison method capable of reducing lock usage according to claim 7, characterized in that: In step S4, the first 10 bits of the last two bytes of each data record are used as a hash value, and the data are connected in series using the row_ind structure according to the hash value.

9. A database table comparison system capable of reducing lock usage, comprising: The export thread includes a source-side export thread and a target-side export thread, which are used to set a first local Hash linked list for each source-side export thread and a target-side export thread respectively, and hash the exported data into their respective first local Hash linked lists according to a hash method. The export thread also sets two global Hash tables for each database table, corresponding to the source and target ends respectively. During the data export process, when a memory block block in the local Hash linked list is full, the memory block is inserted into the corresponding linked list of the global Hash table on its own side. When the total number of memory blocks in a pair of linked lists in the global Hash table exceeds a limit block, a comparison request task is generated, requesting the comparison thread to compare the corresponding linked lists. After the entire export task is completed, a final comparison request task is submitted for each pair of sub-linked lists of the global Hash table on the source and target ends. In the export thread, when a memory block block in the first local Hash linked list is full and the memory block is inserted into the corresponding linked list of the global Hash table on its own side, the corresponding global Hash table is locked. The comparison thread is used to, when receiving a comparison request task, take out the source and target sub-linked lists corresponding to the Hash value in the comparison request task from the global Hash table of the source and target ends respectively, connect each data record in the sub-linked list in series according to the Hash value using the row_ind structure, and perform deduplication calculation. When each deduplication calculation is completed, the comparison thread tries again to get the sub-linked list with the same Hash value as the comparison request task from the global Hash table. If new data is obtained, the deduplication calculation continues. If there is no new data, the remaining data is put back into the global Hash table of the source and target ends; when the calculated residual memory block exceeds a specific value, the data is written to the disk file corresponding to the Hash value.

Images