Batch efficient data comparison method and device based on double-pointer mechanism

Abstract

The invention provides a batch efficient data comparison method and device based on a double-pointer mechanism. The method comprises the steps that data are obtained from a source end database and a target end database in batches; creating a source end pointer and a target end pointer, wherein the source end pointer and the target end pointer respectively correspond to the primary key identifiers of the source end data and the target end data; obtaining all batches of data and sorting the data from small to large according to the feature primary key values; respectively pointing the source end pointer and the target end pointer to a first piece of data in the first batch of data of the source end data and the target end data; and comparing the feature primary key values of the data pointed by the source end pointer and the target end pointer, processing the data according to a comparison result, and moving the pointers until all data of all batches are traversed. According to the method, the main problem of proofreading after data synchronization is solved by adopting a double-pointer technology and combining with a batch processing mode, the comparison frequency is reduced as much as possible, the comparison efficiency is improved, and the dependence on high-end hardware is effectively reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data comparison, and in particular to a batch efficient data comparison method and device based on a double pointer mechanism. Background Art

[0002] With the booming development of 5G technology and the big data industry, technology-driven business and data-driven business strategies have become the mainstream trend. The data era has not only brought about a sharp increase in the amount of data, but also put forward higher requirements for data management, governance and effective use. Among them, ensuring the accuracy of data during data transmission has become a challenge that needs to be solved urgently. However, traditional data comparison devices have shown limitations when dealing with large amounts of data.

[0003] The first typical comparison device completely obtains and stores the data on the source and target sides, and then checks the target side data one by one. Although this device can accurately identify missing or redundant data, memory limitations and traversal time costs become significant bottlenecks when processing large-scale data sets.

[0004] The second device uses Bloom filters to optimize storage and query efficiency by first adding the data from the data source to the Bloom filter, and then checking whether the target data exists in the filter one by one to determine the data accuracy. This method effectively alleviates memory pressure and shortens processing time, but due to the characteristics of Bloom filters, its comparison results may be insufficient in scenarios with high accuracy requirements. Summary of the invention

[0005] To this end, an object of the present invention is to propose a batch efficient data comparison method and device based on a dual pointer mechanism to solve the problems mentioned in the background technology and overcome the deficiencies in the prior art.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a batch efficient data comparison method based on a dual pointer mechanism, comprising:

[0008] Obtain data from the source database and the target database in batches;

[0009] Creating a source end pointer and a target end pointer, wherein the source end pointer and the target end pointer correspond to primary key identifiers of source end data and target end data respectively;

[0010] Get all batch data and sort them in ascending order based on the feature primary key value;

[0011] Point the source pointer and the target pointer to the first data in the first batch of source data and target data respectively;

[0012] Compare the characteristic primary key values ​​of the data pointed to by the source pointer and the target pointer, process the data according to the comparison result, and move the pointer until all the data in all batches are traversed.

[0013] Preferably, comparing the characteristic primary key values ​​of the data pointed to by the source end pointer and the target end pointer, and processing the data according to the comparison result comprises:

[0014] If the characteristic primary key value of the data pointed to by the source-side pointer is greater than the characteristic primary key value of the data pointed to by the target-side pointer, the target-side data pointed to by the target-side pointer is put into the deletion queue;

[0015] If the characteristic primary key value of the data pointed to by the source end pointer is less than the characteristic primary key value of the data pointed to by the target end pointer, the source end data pointed to by the source end pointer is placed in the supplementary queue;

[0016] If the characteristic primary key value of the source pointer is equal to the characteristic primary key value of the target pointer, compare whether the data values ​​pointed to by the source pointer and the target pointer are the same; if different, put the source data pointed to by the source pointer into the supplement queue, and put the target data pointed to by the target pointer into the deletion queue.

[0017] Preferably, the moving pointer comprises:

[0018] When the characteristic primary key value of the data pointed to by the source-side pointer is greater than the characteristic primary key value of the data pointed to by the target-side pointer, the target-side pointer moves down;

[0019] When the characteristic primary key value of the data pointed to by the source pointer is smaller than the characteristic primary key value of the data pointed to by the target pointer, the source pointer moves down;

[0020] When the characteristic primary key value of the source-side pointer is equal to the characteristic primary key value of the target-side pointer, the target-side pointer and the source-side pointer move down at the same time.

[0021] Preferably, the method further comprises:

[0022] During the pointer downward movement process, determine whether the source end pointer and / or the target end pointer has reached the last data of the current batch;

[0023] If the source pointer reaches the last data of the current batch, and the characteristic primary key value of the source pointer is not equal to the characteristic primary key value of the target pointer, the source data pointed to by the source pointer is placed in the supplement queue, and the target data pointed to by the target pointer and the remaining data are placed in the deletion queue;

[0024] If the target pointer reaches the last data of the current batch, and the characteristic primary key value of the source pointer is not equal to the characteristic primary key value of the target pointer, the source data pointed to by the source pointer and the remaining data are placed in the supplement queue, and the target data pointed to by the target pointer is placed in the deletion queue.

[0025] Preferably, the method further comprises:

[0026] If the source pointer reaches the last piece of data in the current batch, and the characteristic primary key value of the source pointer is equal to the characteristic primary key value of the target pointer, the remaining data on the target side will be put into the deletion queue;

[0027] If the target end pointer reaches the last data of the current batch, and the characteristic primary key value of the source end pointer is equal to the characteristic primary key value of the target end pointer, the remaining data of the source end is placed in the supplementary queue.

[0028] Preferably, the method further comprises:

[0029] If the source pointer and / or the target pointer reaches the last data of the current batch, it is determined whether the current batch is the last batch of data. If so, the comparison ends after the last data of the current batch is compared. Otherwise, the source pointer and the target pointer are moved to the first data in the next batch of data.

[0030] Preferably, the determining whether the source end pointer and / or the target end pointer reaches the last piece of data in the current batch comprises:

[0031] The characteristic primary key value of the last data in each batch of source data and target data is recorded respectively. After each movement of the source pointer and / or target pointer, the characteristic primary key value of the source pointer and / or target pointer is compared with the characteristic primary key value of the last data in the current batch of source data and / or target data. If they are equal, it indicates that the source pointer and / or target pointer has reached the last data in the current batch.

[0032] Preferably, the method further comprises:

[0033] The data obtained from the source database and the target database are stored in batches into ordered sets, including the primary key identifiers involved, and multiple fields or full-column fields are specified, where the field data values ​​are processed into corresponding hash values.

[0034] Preferably, the method further comprises:

[0035] Update the source database to the target, record detailed information about the data update, including the time of the update, the updated field, and the values ​​before and after the update, and delete redundant data from the target database.

[0036] In a second aspect, the present invention provides a batch efficient data comparison device based on a dual pointer mechanism, which is applicable to the above-mentioned batch efficient data comparison method based on a dual pointer mechanism, comprising:

[0037] A data acquisition module is configured to acquire data from a source database and a target database in batches respectively;

[0038] A dual pointer module is configured to create a source end pointer and a target end pointer, wherein the source end pointer and the target end pointer correspond to primary key identifiers of source end data and target end data respectively;

[0039] A data preparation module is configured to obtain all batches of data and sort them in ascending order according to the characteristic primary key value, and point the source end pointer and the target end pointer to the first data in the first batch of source end data and target end data respectively;

[0040] The data comparison module compares the characteristic primary key values ​​of the data pointed to by the source pointer and the target pointer, processes the data according to the comparison results, and moves the pointer until all the data in all batches are traversed.

[0041] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of a batch efficient data comparison method based on a dual pointer mechanism as described above are implemented.

[0042] In a fourth aspect, the present invention provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a batch efficient data comparison method based on a dual pointer mechanism as described above.

[0043] Therefore, the present invention has the following beneficial effects:

[0044] Under the condition of limited hardware resources, the method and device of the present invention adopt dual pointer technology and combine it with batch processing to solve the main problem of proofreading after data synchronization. The batch efficient data comparison based on the dual pointer mechanism reduces the number of comparisons as much as possible, speeds up the comparison efficiency, effectively reduces the dependence on high-end hardware, and reduces the requirements of hardware resources while completing accurate data comparison tasks.

[0045] The present invention realizes item-by-item comparison of ordered series based on the database storage format and supports feature primary key sorting, and obtains redundant and missing data through one traversal, thereby significantly improving processing efficiency.

[0046] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0048] Figure 1 It is the overall flow chart of the method of the present invention;

[0049] Figure 2 is a schematic diagram of a data interaction process according to an embodiment of the present invention;

[0050] Figure 3 is a schematic diagram of a data preparation process according to an embodiment of the present invention;

[0051] Figure 4 is a data comparison flow chart of an embodiment of the present invention;

[0052] Figure 5 is a schematic diagram of the structure of a device according to an embodiment of the present invention;

[0053] Figure 6 It is a schematic diagram of the structure of an electronic / computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0055] Aiming at the data verification demand generated after the synchronization of large quantities of data generated in the context of big data, and the practical problems such as hardware requirements, efficiency, and function derived from the excessive amount of data, the present invention proposes a batch efficient data comparison method and device based on a double pointer mechanism.

[0056] The shortcomings of current technology are mainly reflected in the following aspects:

[0057] Inefficiency in processing large-scale data: When faced with massive amounts of data, the time required to achieve accurate data comparison increases significantly, resulting in a slow overall processing process.

[0058] High hardware resource requirements: In order to complete accurate data comparison tasks, existing technologies often need to rely on high-performance hardware devices, which increases the cost and complexity of implementation.

[0059] The trade-off between accuracy and hardware requirements: If a device with lower hardware requirements is used, although the hardware cost can be reduced, the accuracy of some data will often have to be sacrificed to meet efficiency or cost requirements.

[0060] In response to the above problems, the present invention proposes an innovative solution that focuses on processing large-scale data sets ranging from TB to PB. By adopting dual pointer technology and combining batch processing, the present invention achieves significant improvement in processing efficiency while maintaining data comparison accuracy. This device effectively reduces the dependence on high-end hardware and provides new possibilities for efficient data comparison in a big data environment.

[0061] like Figure 1 As shown, the present invention provides a batch efficient data comparison method based on a dual pointer mechanism, comprising:

[0062] Obtain data from the source database and the target database in batches;

[0063] Creating a source end pointer and a target end pointer, wherein the source end pointer and the target end pointer correspond to primary key identifiers of source end data and target end data respectively;

[0064] Get all batch data and sort them in ascending order based on the feature primary key value;

[0065] Point the source pointer and the target pointer to the first data in the first batch of source data and target data respectively;

[0066] Compare the characteristic primary key values ​​of the data pointed to by the source pointer and the target pointer, process the data according to the comparison result, and move the pointer until all the data in all batches are traversed.

[0067] Under the condition of limited hardware resources, the method and device of the present invention adopt dual pointer technology and combine it with batch processing to solve the main problem of proofreading after data synchronization. The batch efficient data comparison based on the dual pointer mechanism reduces the number of comparisons as much as possible, speeds up the comparison efficiency, effectively reduces the dependence on high-end hardware, and reduces the requirements of hardware resources while completing accurate data comparison tasks.

[0068] The present invention realizes item-by-item comparison of ordered series based on the database storage format and supports feature primary key sorting, and obtains redundant and missing data through one traversal, thereby significantly improving processing efficiency.

[0069] Furthermore, comparing the characteristic primary key values ​​of the data pointed to by the source end pointer and the target end pointer, and processing the data according to the comparison result includes:

[0070] If the characteristic primary key value of the data pointed to by the source-side pointer is greater than the characteristic primary key value of the data pointed to by the target-side pointer, the target-side data pointed to by the target-side pointer is put into the deletion queue;

[0071] If the characteristic primary key value of the data pointed to by the source end pointer is less than the characteristic primary key value of the data pointed to by the target end pointer, the source end data pointed to by the source end pointer is placed in the supplementary queue;

[0072] If the characteristic primary key value of the source pointer is equal to the characteristic primary key value of the target pointer, compare whether the data values ​​pointed to by the source pointer and the target pointer are the same; if different, put the source data pointed to by the source pointer into the supplement queue, and put the target data pointed to by the target pointer into the deletion queue.

[0073] Further, the moving pointer includes:

[0074] When the characteristic primary key value of the data pointed to by the source-side pointer is greater than the characteristic primary key value of the data pointed to by the target-side pointer, the target-side pointer moves down;

[0075] When the characteristic primary key value of the data pointed to by the source pointer is smaller than the characteristic primary key value of the data pointed to by the target pointer, the source pointer moves down;

[0076] When the characteristic primary key value of the source-side pointer is equal to the characteristic primary key value of the target-side pointer, the target-side pointer and the source-side pointer move down at the same time.

[0077] Furthermore, the method further comprises:

[0078] During the pointer downward movement process, determine whether the source end pointer and / or the target end pointer has reached the last data of the current batch;

[0079] If the source pointer reaches the last data of the current batch, and the characteristic primary key value of the source pointer is not equal to the characteristic primary key value of the target pointer, the source data pointed to by the source pointer is placed in the supplement queue, and the target data pointed to by the target pointer and the remaining data are placed in the deletion queue;

[0080] If the target pointer reaches the last data of the current batch, and the characteristic primary key value of the source pointer is not equal to the characteristic primary key value of the target pointer, the source data pointed to by the source pointer and the remaining data are placed in the supplement queue, and the target data pointed to by the target pointer is placed in the deletion queue.

[0081] Furthermore, the method further comprises:

[0082] If the source pointer reaches the last piece of data in the current batch, and the characteristic primary key value of the source pointer is equal to the characteristic primary key value of the target pointer, the remaining data on the target side will be put into the deletion queue;

[0083] If the target end pointer reaches the last data of the current batch, and the characteristic primary key value of the source end pointer is equal to the characteristic primary key value of the target end pointer, the remaining data of the source end is placed in the supplementary queue.

[0084] Furthermore, the method further comprises:

[0085] If the source pointer and / or the target pointer reaches the last data of the current batch, it is determined whether the current batch is the last batch of data. If so, the comparison ends after the last data of the current batch is compared. Otherwise, the source pointer and the target pointer are moved to the first data in the next batch of data.

[0086] Further, the determining whether the source end pointer and / or the target end pointer reaches the last piece of data in the current batch includes:

[0087] The characteristic primary key value of the last data in each batch of source data and target data is recorded respectively. After each movement of the source pointer and / or target pointer, the characteristic primary key value of the source pointer and / or target pointer is compared with the characteristic primary key value of the last data in the current batch of source data and / or target data. If they are equal, it indicates that the source pointer and / or target pointer has reached the last data in the current batch.

[0088] Furthermore, the method further comprises:

[0089] The data obtained from the source database and the target database are stored in batches into ordered sets, including the primary key identifiers involved, and multiple fields or full-column fields are specified, where the field data values ​​are processed into corresponding hash values.

[0090] Furthermore, the method further comprises:

[0091] Update the source database to the target, record detailed information about the data update, including the time of the update, the updated field, and the values ​​before and after the update, and delete redundant data from the target database.

[0092] In one embodiment, Figure 2 As shown in the figure, the data flow path is depicted in detail, including shared communication through the "feature primary key value", the data items pointed to by the "source pointer" and "destination pointer" in their respective databases, and the roles of the two special structures "supplement queue" and "deletion queue", which are used to handle additional data addition and deletion requests respectively.

[0093] Specifically, data preparation is first performed. The data preparation process is shown as follows: Figure 3 As shown, data with N records are obtained in batches from the source database table and the target database table (ie, the data tables in the source database and the target database), respectively, where N is a positive integer greater than or equal to 1.

[0094] The source data and the target data, each record data has a primary key identifier, and each record data includes specified multiple columns or full column fields.

[0095] The source data and the target data are stored in ordered sets, including the primary key identifiers involved, and multiple fields or full-column fields are specified, where the field data values ​​are processed as corresponding hash values. After all fields are processed as hash values, field matching can be performed to compare whether the hash values ​​corresponding to the source and target field values ​​to be matched are the same.

[0096] Create a target-side pointer and a source-side pointer respectively, corresponding to the above-mentioned source-side and target-side primary key identifiers.

[0097] Secondly, initialization and data acquisition are performed. According to the order of feature primary keys (such as ID) from small to large, the first batch of data is obtained from the data source table (source and target), and the feature primary key value of the last data of the current batch in the source and target tables is recorded respectively.

[0098] Then put the data into the comparison array, put the obtained data in sequence into the comparison array of the corresponding length of the current batch, and save the feature primary key value at the first position of the comparison array as the source end pointer and the target end pointer respectively.

[0099] Data is stored in different categories and placed in different storage containers according to the detailed requirements of data comparison.

[0100] As an implementation method, you can compare only the number of entries: only obtain the primary key value of the feature, without obtaining other data

[0101] As an implementation method, the comparison may be performed field by field: a comparison container is created according to the selected fields, and the field value and the feature primary key are stored together in a comparison array.

[0102] As an implementation method, special data types are compared: for files, pictures, videos, etc., hash values ​​are calculated and stored using a hash algorithm (such as MD5).

[0103] Next, we compare the data and compare the characteristic primary key values ​​of the source pointer and the target pointer. The comparison results include:

[0104] The data is completely consistent, that is, the data values ​​in the comparison container are exactly the same, then the data is determined to be completely consistent;

[0105] If the data is inconsistent, that is, the data values ​​are different, the target table data synchronization error is determined, and the source data is placed in the supplement queue and the target data is placed in the deletion queue.

[0106] If the feature primary key values ​​are different, they are processed according to the following rules:

[0107] Target-side redundancy: If the primary key value of the source-side feature is greater than that of the target-side, the target-side data is put into the deletion queue;

[0108] Missing on the target side: If the primary key value of the source feature is smaller than that of the target, the source data is placed in the supplementary queue.

[0109] Move the pointer based on the comparison results, including:

[0110] The primary keys are the same: the source pointer and the target pointer move down at the same time;

[0111] The source end is greater than the target end: the target end pointer moves down;

[0112] The source end is smaller than the target end: the source end pointer moves down;

[0113] End of batch: If the current feature primary key value is the last data in this batch, get the next batch of data and put the pointer back to the first position;

[0114] End of source data: If the source pointer reaches the end of the data when moving down, the remaining target data is put into the deletion queue;

[0115] End of target data: If the target pointer reaches the end of the data when moving down, the remaining source data is placed in the supplementary queue.

[0116] If the source-side feature primary key value and the target-side feature primary key value both reach the end of their respective data ends of the batch when the pointer moves downward, the next batch of data is compared until the last batch of comparisons is completed.

[0117] In one embodiment, Figure 4 As shown, the data comparison of the present invention is explained in the form of a step flow.

[0118] S400: Start comparison:

[0119] S401: Compare the source end pointer and the target end pointer to see whether their characteristic primary key values ​​are equal, and make a judgment;

[0120] If yes, execute S402, otherwise, execute S406;

[0121] S402: Compare the data values ​​pointed to by the source end pointer and the target end pointer of the current batch, and make a judgment;

[0122] If the data values ​​pointed to by the source end pointer and the target end pointer of the current batch are the same, execute S404, otherwise execute S403;

[0123] S403: putting the source data pointed to by the source pointer into the supplement queue, and putting the target data pointed to by the target pointer into the deletion queue, and proceeding to S404;

[0124] S404: Determine whether the source end pointer and / or the target end pointer reaches the last data of the current batch;

[0125] If yes, execute S413; otherwise execute S405;

[0126] S405: Move the source end pointer and the target end pointer downward simultaneously, and return to execute S401;

[0127] S406: Determine whether the source end pointer and / or the target end pointer is the last data of the current batch;

[0128] If yes, execute S410, otherwise execute S407;

[0129] S407: Compare the characteristic primary key values ​​of the source end pointer and the target end pointer, and make a judgment;

[0130] If the characteristic primary key value of the source end pointer is less than the characteristic primary key value of the target end pointer, execute S409, otherwise execute 408;

[0131] S408: putting the target end data pointed to by the target end pointer into the deletion queue, moving the target end pointer downward, and returning to execute S401;

[0132] S409: Put the source data pointed to by the source pointer into the supplementary queue, move the source pointer downward, and return to execute S401;

[0133] S410: Determine whether the source data ends (i.e., whether the source pointer reaches the end of the source data when it moves down). If so, execute S412; otherwise, execute S411;

[0134] S411: put the source data pointed to by the source pointer and the remaining data into the supplement queue, put the target data pointed to by the target pointer into the deletion queue, and go to S416;

[0135] S412: putting the source data pointed to by the source pointer into the supplement queue, and putting the target data pointed to by the target pointer and the remaining data into the deletion queue, and proceeding to S416;

[0136] S413: Determine whether the source data ends (i.e., whether the source pointer reaches the end of the source data when it moves down). If so, execute S415; otherwise, execute S414;

[0137] S414: Put the remaining data on the source end into a supplementary queue and proceed to S416;

[0138] S415: Put the remaining data on the target end into a supplementary queue and proceed to S416;

[0139] S416: Determine whether the last batch of data has been reached;

[0140] If yes, execute S418; otherwise execute S417;

[0141] S417: Get the next batch of data, move the source end pointer and the target end pointer to the first position, and return to execute S401;

[0142] S418: The comparison ends.

[0143] After the comparison is completed, update the target data: update the data on the source side to the target side to maintain the consistency between the two data sets, that is, update the data in the supplementary queue to the target side. This usually involves executing an UPDATE statement in the target database.

[0144] Record update log: record detailed information of data update, including update time, updated fields, values ​​before and after update, etc., for subsequent audit and tracking.

[0145] Delete redundant data: Delete the redundant data from the target database, that is, delete the data in the deletion queue from the target database.

[0146] The present invention sorts the data according to the feature primary key value, splits the data into several parts according to the order, performs comparison processing in batches, traverses the data of individual batches, filters out the data that does not meet the requirements and saves them separately, and performs statistics on the results after all processing is completed.

[0147] The present invention solves the main problem of proofreading after data synchronization under the condition of limited hardware resources, reduces the number of comparisons as much as possible, and improves the comparison efficiency.

[0148] The present invention realizes item-by-item comparison of ordered series based on the database storage format and supporting feature primary key sorting, and obtains redundant and missing data through one traversal.

[0149] The present invention also has the following advantages:

[0150] Support sampling / full data comparison;

[0151] Supports bidirectional data comparison between source and target ends, filtering out bidirectional data problems;

[0152] Meet the data synchronization requirements in the current big data environment, as well as the data verification and operation and maintenance work after synchronization;

[0153] Data comparison can be achieved without affecting the use of the production environment.

[0154] The above primary key identifier represents the field item corresponding to the primary key in the table (the corresponding data table in the database), the field item corresponding to the unique constraint, and the field item corresponding to the unique index.

[0155] Characteristic primary key: A primary key field in a data table that is not empty, cannot be repeated, and is ordered.

[0156] like Figure 5 As shown, the present invention provides a batch efficient data comparison device based on a dual pointer mechanism, which is applicable to the above-mentioned batch efficient data comparison method based on a dual pointer mechanism, including:

[0157] A data acquisition module is configured to acquire data from a source database and a target database in batches respectively;

[0158] A dual pointer module is configured to create a source end pointer and a target end pointer, wherein the source end pointer and the target end pointer correspond to primary key identifiers of source end data and target end data respectively;

[0159] A data preparation module is configured to obtain all batches of data and sort them in ascending order according to the characteristic primary key value, and point the source end pointer and the target end pointer to the first data in the first batch of source end data and target end data respectively;

[0160] The data comparison module compares the characteristic primary key values ​​of the data pointed to by the source pointer and the target pointer, processes the data according to the comparison results, and moves the pointer until all the data in all batches are traversed.

[0161] Furthermore, comparing the characteristic primary key values ​​of the data pointed to by the source end pointer and the target end pointer, and processing the data according to the comparison result includes:

[0162] If the characteristic primary key value of the data pointed to by the source-side pointer is greater than the characteristic primary key value of the data pointed to by the target-side pointer, the target-side data pointed to by the target-side pointer is put into the deletion queue;

[0163] If the characteristic primary key value of the data pointed to by the source end pointer is less than the characteristic primary key value of the data pointed to by the target end pointer, the source end data pointed to by the source end pointer is placed in the supplementary queue;

[0164] If the characteristic primary key value of the source pointer is equal to the characteristic primary key value of the target pointer, compare whether the data values ​​pointed to by the source pointer and the target pointer are the same; if different, put the source data pointed to by the source pointer into the supplement queue, and put the target data pointed to by the target pointer into the deletion queue.

[0165] Further, the moving pointer includes:

[0166] When the characteristic primary key value of the data pointed to by the source-side pointer is greater than the characteristic primary key value of the data pointed to by the target-side pointer, the target-side pointer moves down;

[0167] When the characteristic primary key value of the data pointed to by the source pointer is smaller than the characteristic primary key value of the data pointed to by the target pointer, the source pointer moves down;

[0168] When the characteristic primary key value of the source-side pointer is equal to the characteristic primary key value of the target-side pointer, the target-side pointer and the source-side pointer move down at the same time.

[0169] Furthermore, it also includes:

[0170] During the pointer downward movement process, determine whether the source end pointer and / or the target end pointer has reached the last data of the current batch;

[0171] If the source pointer reaches the last data of the current batch, and the characteristic primary key value of the source pointer is not equal to the characteristic primary key value of the target pointer, the source data pointed to by the source pointer is placed in the supplement queue, and the target data pointed to by the target pointer and the remaining data are placed in the deletion queue;

[0172] If the target pointer reaches the last data of the current batch, and the characteristic primary key value of the source pointer is not equal to the characteristic primary key value of the target pointer, the source data pointed to by the source pointer and the remaining data are placed in the supplement queue, and the target data pointed to by the target pointer is placed in the deletion queue.

[0173] Furthermore, it also includes:

[0174] If the source pointer reaches the last piece of data in the current batch, and the characteristic primary key value of the source pointer is equal to the characteristic primary key value of the target pointer, the remaining data on the target side will be put into the deletion queue;

[0175] If the target end pointer reaches the last data of the current batch, and the characteristic primary key value of the source end pointer is equal to the characteristic primary key value of the target end pointer, the remaining data of the source end is placed in the supplementary queue.

[0176] Furthermore, it also includes:

[0177] If the source pointer and / or the target pointer reaches the last data of the current batch, it is determined whether the current batch is the last batch of data. If so, the comparison ends after the last data of the current batch is compared. Otherwise, the source pointer and the target pointer are moved to the first data in the next batch of data.

[0178] Further, the determining whether the source end pointer and / or the target end pointer reaches the last piece of data in the current batch includes:

[0179] The characteristic primary key value of the last data in each batch of source data and target data is recorded respectively. After each movement of the source pointer and / or target pointer, the characteristic primary key value of the source pointer and / or target pointer is compared with the characteristic primary key value of the last data in the current batch of source data and / or target data. If they are equal, it indicates that the source pointer and / or target pointer has reached the last data in the current batch.

[0180] Furthermore, it also includes:

[0181] The data obtained from the source database and the target database are stored in batches into ordered sets, including the primary key identifiers involved, and multiple fields or full-column fields are specified, where the field data values ​​are processed into corresponding hash values.

[0182] Furthermore, it also includes:

[0183] Update the source database to the target, record detailed information about the data update, including the time of the update, the updated field, and the values ​​before and after the update, and delete redundant data from the target database.

[0184] To solve the above technical problems, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of a batch efficient data comparison method based on a dual pointer mechanism as described above are implemented.

[0185] like Figure 6 As shown, the electronic device / computer device includes a memory, a processor, and a network interface that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device with component memory, a processor, a network interface, and an operating system, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0186] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0187] The memory may have one or more memories, and may include at least one type of readable storage medium, the readable storage medium including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device. Of course, the memory may also include both an internal storage unit of a computer device and an external storage device thereof. In this embodiment, the memory is generally used to store an operating system and various application software installed on the computer device, such as a program code of a batch efficient data comparison method based on a dual pointer mechanism, etc. In addition, the memory may also be used to temporarily store various types of data that have been output or are to be output.

[0188] The processor can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chips in some embodiments. The processor is generally used to control the overall operation of a computer device. In the present embodiment, the processor is used to run the program code stored in the memory or process data, such as running the program code of the batch efficient data comparison method based on the dual pointer mechanism.

[0189] The network interface may include a wireless network interface and / or a wired network interface, which is generally used to establish a communication connection between a computer device and other electronic devices.

[0190] The present invention also provides another embodiment, that is, providing a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned batch efficient data comparison method based on a dual pointer mechanism.

[0191] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0192] It is not difficult for those skilled in the art to understand that the present invention includes any combination of the invention content and specific implementation methods of the above specification and the various parts shown in the drawings. Due to the limited space and to make the specification concise, the various schemes composed of these combinations are not described one by one. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0193] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. Those skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and purpose of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A batch efficient data comparison method based on a dual pointer mechanism, characterized in that: include: Obtain data from the source database and the target database in batches; Creating a source end pointer and a target end pointer, wherein the source end pointer and the target end pointer correspond to primary key identifiers of source end data and target end data respectively; Get all batch data and sort them in ascending order based on the feature primary key value; Point the source pointer and the target pointer to the first data in the first batch of source data and target data respectively; Compare the characteristic primary key values ​​of the data pointed to by the source pointer and the target pointer, process the data according to the comparison result, and move the pointer until all the data in all batches are traversed.

2. The batch efficient data comparison method based on the dual pointer mechanism according to claim 1 is characterized in that: The comparing the characteristic primary key values ​​of the data pointed to by the source end pointer and the target end pointer, and processing the data according to the comparison result includes: If the characteristic primary key value of the data pointed to by the source-side pointer is greater than the characteristic primary key value of the data pointed to by the target-side pointer, the target-side data pointed to by the target-side pointer is put into the deletion queue; If the characteristic primary key value of the data pointed to by the source end pointer is less than the characteristic primary key value of the data pointed to by the target end pointer, the source end data pointed to by the source end pointer is placed in the supplementary queue; If the characteristic primary key value of the source pointer is equal to the characteristic primary key value of the target pointer, compare whether the data values ​​pointed to by the source pointer and the target pointer are the same; if different, put the source data pointed to by the source pointer into the supplement queue, and put the target data pointed to by the target pointer into the deletion queue.

3. The batch efficient data comparison method based on the dual pointer mechanism according to claim 2 is characterized in that: The moving pointers include: When the characteristic primary key value of the data pointed to by the source-side pointer is greater than the characteristic primary key value of the data pointed to by the target-side pointer, the target-side pointer moves down; When the characteristic primary key value of the data pointed to by the source pointer is smaller than the characteristic primary key value of the data pointed to by the target pointer, the source pointer moves down; When the characteristic primary key value of the source-side pointer is equal to the characteristic primary key value of the target-side pointer, the target-side pointer and the source-side pointer move down at the same time.

4. The batch efficient data comparison method based on the dual pointer mechanism according to claim 3 is characterized in that: Also includes: During the pointer downward movement process, determine whether the source end pointer and / or the target end pointer has reached the last data of the current batch; If the source pointer reaches the last data of the current batch, and the characteristic primary key value of the source pointer is not equal to the characteristic primary key value of the target pointer, the source data pointed to by the source pointer is placed in the supplement queue, and the target data pointed to by the target pointer and the remaining data are placed in the deletion queue; If the target pointer reaches the last data of the current batch, and the characteristic primary key value of the source pointer is not equal to the characteristic primary key value of the target pointer, the source data pointed to by the source pointer and the remaining data are placed in the supplement queue, and the target data pointed to by the target pointer is placed in the deletion queue.

5. The method for batch efficient data comparison based on a dual pointer mechanism according to claim 4 is characterized in that: Also includes: If the source pointer reaches the last piece of data in the current batch, and the characteristic primary key value of the source pointer is equal to the characteristic primary key value of the target pointer, the remaining data on the target side will be put into the deletion queue; If the target end pointer reaches the last data of the current batch, and the characteristic primary key value of the source end pointer is equal to the characteristic primary key value of the target end pointer, the remaining data of the source end is placed in the supplementary queue.

6. The method for batch efficient data comparison based on a dual pointer mechanism according to claim 4, characterized in that: Also includes: If the source pointer and / or the target pointer reaches the last data of the current batch, it is determined whether the current batch is the last batch of data. If so, the comparison ends after the last data of the current batch is compared. Otherwise, the source pointer and the target pointer are moved to the first data in the next batch of data.

7. The method for batch efficient data comparison based on a dual pointer mechanism according to claim 4, characterized in that: The step of determining whether the source end pointer and / or the target end pointer reaches the last piece of data in the current batch includes: The characteristic primary key value of the last data in each batch of source data and target data is recorded respectively. After each movement of the source pointer and / or target pointer, the characteristic primary key value of the source pointer and / or target pointer is compared with the characteristic primary key value of the last data in the current batch of source data and / or target data. If they are equal, it indicates that the source pointer and / or target pointer has reached the last data in the current batch.

8. The batch efficient data comparison method based on the dual pointer mechanism according to claim 1 is characterized in that: Also includes: The data obtained from the source database and the target database are stored in batches into ordered sets, including the primary key identifiers involved, and multiple fields or full-column fields are specified, where the field data values ​​are processed into corresponding hash values.

9. A batch efficient data comparison method based on a dual pointer mechanism according to any one of claims 1 to 8, characterized in that: Also includes: Update the source database to the target, record detailed information about the data update, including the time of the update, the updated field, and the values ​​before and after the update, and delete redundant data from the target database.

10. A batch efficient data comparison device based on a dual pointer mechanism, applicable to a batch efficient data comparison method based on a dual pointer mechanism as claimed in any one of claims 1 to 9, characterized in that: include: A data acquisition module is configured to acquire data from a source database and a target database in batches respectively; A dual pointer module is configured to create a source end pointer and a target end pointer, wherein the source end pointer and the target end pointer correspond to primary key identifiers of source end data and target end data respectively; A data preparation module is configured to obtain all batches of data and sort them in ascending order according to the characteristic primary key value, and point the source end pointer and the target end pointer to the first data in the first batch of source end data and target end data respectively; The data comparison module is configured to compare the characteristic primary key values ​​of the data pointed to by the source end pointer and the target end pointer, process the data according to the comparison result, and move the pointer until all the data of all batches are traversed.

Images