Data correction and supplement method, device, equipment and system

Abstract

The present invention relates to a method, device, equipment and system for data correction and complementation, and belongs to the technical field of data transmission. The method, device, equipment and device, in the data transmission process between the source end and the target end, determine the source end data and the target end data in real time, and compare the data of the two in real time, determine whether correction is needed through data quality accuracy and preset correction conditions, when correction is needed, the target end restarts the collection thread to collect source end data according to the set recovery checkpoint, thereby realizing the associated operation of the three processes of data transmission, data consistency verification, and abnormal complementation. In the data transmission process, it is judged in real time whether the data transmission is abnormal through the correction conditions. When the data transmission is abnormal, the source end collection offset that needs correction is specified, and the data collection thread is restarted to the offset in time to perform abnormal complementation, which is convenient and fast, and overcomes the current technical problem of poor timeliness of data correction and complementation.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to a data deviation correction and complement method, device, equipment and system. Background Art

[0002] In the Internet age, data transmission has become an important way to support information transmission. In order to ensure the accuracy of data transmission, data consistency verification tools are usually used to determine whether there are omissions or inconsistencies in the transmitted data after the data transmission is completed. That is, data consistency verification is performed after the data transmission is completed. Based on the verification results, data is supplemented or data transmission is performed again to ensure the accuracy of data transmission.

[0003] However, the three processes of data transmission, data consistency check and exception complement in the related technology are independent of each other. After a certain period of time (for example, one day) of data transmission is completed, data consistency check must be performed, and finally exception complement must be performed, which has poor timeliness. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a data correction and complement method, device, equipment and system to overcome the current technical problem of poor timeliness of data correction and complement.

[0005] To achieve the above objectives, the present invention adopts the following technical solutions:

[0006] On the one hand, a data correction and supplement method includes:

[0007] Determining source-end data and target-end data in real time based on a specified checkpoint, wherein the specified checkpoint identifies a starting point for a data consistency check to be performed;

[0008] Taking the initial checkpoint as the first comparison checkpoint, the data quality accuracy of the first comparison checkpoint in the source data and the target data is calculated; wherein, according to the data collection sequence, the initial checkpoint is any data point after the designated checkpoint;

[0009] Based on the data quality accuracy and a preset correction threshold, a threshold of times less than the correction threshold, and a threshold of times of correction retries, determining whether the first comparison checkpoint meets the correction condition;

[0010] Polling and calculating the data quality accuracy of each comparison checkpoint, and determining whether the comparison checkpoint meets the correction condition; the correction condition includes: the data quality accuracy of the comparison checkpoint is less than the correction threshold, and the number of times the data quality accuracy of the comparison checkpoint is less than the correction threshold is greater than or equal to the number of times less than the correction threshold; and the number of correction retries for the same recovery checkpoint is less than or equal to the correction retry number threshold;

[0011] If the comparison checkpoint meets the correction condition, the target end restarts the collection thread to collect the source end data according to the recovery checkpoint, and uses the recovery checkpoint as the designated checkpoint to cyclically determine the source end data and the target end data, and the number of correction retries of the recovery checkpoint is increased by 1; wherein, the recovery checkpoint is any data point in the comparison checkpoint that does not meet the correction condition before the current comparison checkpoint.

[0012] Optionally, the method for calculating the data quality accuracy of any comparison checkpoint in the calculation of the data quality accuracy of each comparison checkpoint includes:

[0013] Determine each source table column data of the source end data, the total number of source table column data, each target table column data of the target end data, and data comparison rules in the comparison checkpoint;

[0014] Calculating the total number of the source table column data and the target table column data that meet the data comparison rule after verification through the data comparison rule;

[0015] The ratio of the total number of the data that meet the data comparison rule to the total number of the source table column data is calculated, and the ratio is used as the data quality accuracy of the comparison checkpoint.

[0016] Optionally, the data comparison rules include: mapping relationship between source table column data and target table column data, data partitioning and consistency requirements;

[0017] The calculating the total number of the source table column data and the target table column data that meet the data comparison rule after verification through the data comparison rule includes:

[0018] According to the data partition, the source table column data is divided into different sub-source table column data, and the target table column data is divided into different sub-target table column data;

[0019] According to the mapping relationship, the sub-source table column data and the corresponding sub-target table column data are compared, and if the degree of similarity between the two meets the consistency requirement, it is determined that the sub-source table column data and the corresponding sub-target table column data meet the data comparison rule;

[0020] Poll and compare each of the sub-source table column data and the corresponding sub-target table column data. When any sub-source table column data and the corresponding sub-target table column data meet the data comparison rule, the number of records that meet the data comparison rule is incremented by 1 to obtain the total number that meets the data comparison rule.

[0021] Optionally, also include:

[0022] When the data quality accuracy of the comparison checkpoint is greater than or equal to the correction threshold, obtaining the current offset of the source data;

[0023] The recovery checkpoint is set according to the current offset.

[0024] Optionally, the determining whether the comparison checkpoint meets the deviation correction condition includes:

[0025] Determine whether the data quality accuracy of the comparison checkpoint is less than the deviation correction threshold;

[0026] When the data quality accuracy of the comparison checkpoint is less than the correction threshold, the number of times the data quality accuracy of the comparison checkpoint is less than the correction threshold is increased by 1, and the number of times the data quality accuracy of the comparison checkpoint is less than the correction threshold is updated;

[0027] Determine whether the number of times the data quality accuracy of the updated comparison checkpoint is less than the correction threshold is greater than or equal to the threshold number of times less than the correction threshold;

[0028] If the number of times the data quality accuracy of the updated comparison checkpoint is less than the correction threshold is greater than or equal to the threshold number of times less than the correction threshold, and the number of retries of the same recovery checkpoint is less than or equal to the correction retry threshold, then it is determined that the comparison checkpoint meets the correction condition.

[0029] Optionally, also include:

[0030] The target end establishes a globally unique index after collecting data from the source end;

[0031] When the target end restarts the collection thread to collect source data according to the recovery checkpoint, it is determined whether the collected source data already exists in the target end data according to the globally unique index; if so, the currently collected source data is skipped; if not, the currently collected source data is written to the target end, and a globally unique index of the currently collected source data is generated.

[0032] Optionally, the deviation correction threshold includes 95%; the threshold of the number of times less than the deviation correction threshold includes 3 times; and the threshold of the number of deviation correction retries includes 9 times.

[0033] In another aspect, a data correction and complement device comprises:

[0034] A determination module, configured to determine source-end data and target-end data in real time based on a specified checkpoint, wherein the specified checkpoint identifies a starting point for a data consistency check to be performed;

[0035] A calculation module, configured to calculate the data quality accuracy of the first comparison checkpoint in the source data and the target data using the initial checkpoint as the first comparison checkpoint; wherein, according to the data collection sequence, the initial checkpoint is any data point after the designated checkpoint;

[0036] A judgment module is used to judge whether the first comparison checkpoint meets the correction condition based on the data quality accuracy and a preset correction threshold, a threshold value of times less than the correction threshold, and a threshold value of times of correction retries; poll and calculate the data quality accuracy of each comparison checkpoint, and judge whether the comparison checkpoint meets the correction condition; the correction condition includes: the data quality accuracy of the comparison checkpoint is less than the correction threshold, and the number of times the data quality accuracy of the comparison checkpoint is less than the correction threshold is greater than or equal to the threshold value of times less than the correction threshold; and the number of correction retries of the same recovery checkpoint is less than or equal to the threshold value of times of correction retries;

[0037] A trigger module is used to enable the target end to restart the collection thread to collect source end data according to the recovery checkpoint if the comparison checkpoint meets the correction condition, and to cyclically determine the source end data and the target end data with the recovery checkpoint as the designated checkpoint, and the number of correction retries of the recovery checkpoint is increased by 1; wherein, the recovery checkpoint is any data point in the comparison checkpoint that does not meet the correction condition before the current comparison checkpoint.

[0038] In another aspect, a data correction and complement device includes a processor and a memory, wherein the processor is connected to the memory:

[0039] Wherein, the processor is used to call and execute the program stored in the memory;

[0040] The memory is used to store the program, and the program is used to execute at least any one of the data correction and complement methods described above.

[0041] On the other hand, a data correction and complement system is characterized in that it includes: a source end, a target end and the data correction and complement device described above; the source end is used to provide source end data; the target end collects source end data of the source end, and when the comparison checkpoint meets the correction condition, restarts the collection thread according to the recovery checkpoint to collect source end data; the data correction and complement device is used to perform correction and complement on the target end.

[0042] The technical solution provided by the present invention has at least the following beneficial effects:

[0043] During the data transmission process between the source and target ends, the source data and the target data are determined in real time, and the data of the two are compared in real time. The data quality accuracy and the preset correction conditions are used to determine whether correction is needed. When correction is needed, the target end restarts the collection thread to collect the source data according to the set recovery checkpoint, thereby realizing the associated operation of the three processes of data transmission, data consistency check, and abnormal complement. Therefore, the technical solution of the present application is adopted. During the data transmission process, whether the data transmission is abnormal is determined in real time through the correction conditions. When the data transmission is abnormal (that is, the data consistency check fails), the source end collection offset that needs correction is specified, and the data collection thread is restarted to the offset in time to perform abnormal complement. It is convenient and fast, and overcomes the current technical problem of poor timeliness of data correction and complement. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 A schematic diagram of the structure of a data correction and complement system provided by one embodiment of the present invention;

[0046] Figure 2 A schematic diagram of the structure of a data deflection correction and complement device provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of a flow chart of a data correction and complement method provided by an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of the structure of a data de-skew correction and complement device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0049] To make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described in detail below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0050] As described in the background technology, the three processes of data transmission, data consistency check and exception complement in the related technology are independent of each other. After a certain period of time (for example, one day) of data transmission is completed, data consistency check must be performed, and finally exception complement must be performed, which has poor timeliness.

[0051] Based on this, the embodiments of the present invention provide a data correction and complement method, device, equipment and system to overcome the current technical problem of poor timeliness of data correction and complement.

[0052] Figure 1 A structural diagram of a data correction and complement system provided by an embodiment of the present invention, see Figure 1 The system provided in this embodiment may include the following structures: a source end 1, a target end 2 and a data correction and complement device 3; the source end is used to provide source end data; the target end collects the source end data of the source end, and when the comparison checkpoint meets the correction condition, restarts the collection thread to collect the source end data according to the recovery checkpoint; the data correction and complement device is used to perform correction and complement on the target end.

[0053] Figure 2 A schematic diagram of the structure of a data de-skew correction and complement device provided in one embodiment of the present invention.

[0054] The data correction and complement device 3 may include a processor 31 and a memory 32 (see Figure 2 ), the processor is connected to the memory:

[0055] The processor is used to call and execute the program stored in the memory;

[0056] A memory is used to store a program, and the program is used to execute at least any one of the data correction and complement methods provided in the present application.

[0057] In a specific data correction and complement implementation process, the source end provides source data; the target end obtains the source data from the source end and writes it into itself to obtain the target end data; the data correction and complement device can be integrated and set in the target end, or it can be set separately. The data correction and complement method can be set in the data correction and complement device. During the data transmission process, the correction conditions are used in real time to determine whether the data transmission is abnormal. When it is detected that the target end needs to perform data correction and complement, the target end is triggered to restart the collection thread according to the recovery checkpoint to collect the source end data, so as to perform abnormal complementation.

[0058] It can be understood that by adopting the technical solution provided in this embodiment, during the data transmission process, it is determined in real time whether the data transmission is abnormal through the correction conditions. When the data transmission is abnormal (that is, the data consistency check fails), the data acquisition thread is restarted in time to perform abnormal complementation, which is convenient and fast, and overcomes the current technical problem of poor timeliness of data correction and complementation.

[0059] Based on a general inventive concept, an embodiment of the present invention provides a data de-skew correction and complement method.

[0060] Figure 3 A flowchart of a data correction and complement method provided by an embodiment of the present invention is shown in FIG. Figure 3 The technical solution provided in this embodiment may include the following steps:

[0061] Step S31, determining the source end data and the target end data in real time based on the designated checkpoint, wherein the designated checkpoint identifies the starting point for the data consistency check to be performed.

[0062] It is worth noting that the source data and the target data correspond one to one, and a designated checkpoint may be a data point in the source data and a data point in the corresponding target data.

[0063] Specifically, during the data transmission process between the source and target ends, a certain data point can be initialized as a designated checkpoint to obtain the data of the source and target ends, and the user can set it according to needs.

[0064] Step S32, taking the initial checkpoint as the first comparison checkpoint, calculating the data quality accuracy of the first comparison checkpoint in the source data and the target data; wherein, according to the data collection sequence, the initial checkpoint is any data point after the specified checkpoint.

[0065] During the correction process, the initial checkpoint, recovery checkpoint and comparison checkpoint can be initialized initially.

[0066] In the process of real-time data acquisition, the user can define any data point after the specified checkpoint as the initial checkpoint according to the data collection order. For example, the next data point after the checkpoint can be specified as the initial checkpoint to improve the comprehensiveness of the inspection, thereby calculating the data quality accuracy of the comparison checkpoint.

[0067] In some embodiments, a method for calculating the data quality accuracy of any comparison checkpoint in calculating the data quality accuracy of each comparison checkpoint may specifically include the following steps:

[0068] Determine each source table column data of the source end data in the comparison checkpoint, the total number of source table column data, each target table column data of the target end data, and data comparison rules;

[0069] Calculate the total number of source table column data and target table column data that meet the data comparison rules after verification through the data comparison rules;

[0070] Calculate the ratio of the total number of data that meet the data comparison rules to the total number of source table column data, and use the ratio as the data quality accuracy of the comparison checkpoint.

[0071] The data point may be table column data, the comparison checkpoint in the source end data is the source table column data, and the comparison checkpoint in the target end data is the target table column data.

[0072] It can be understood that by comparing each source table column data with the target table column data, the accuracy of the data quality precision calculation is improved.

[0073] In some embodiments, the data comparison rules include: mapping relationships between source table column data and target table column data, data partitioning, and consistency requirements;

[0074] Calculates the total number of source table column data and target table column data that meet the data comparison rules after verification, including:

[0075] According to the data partition, the source table column data is divided into different sub-source table column data, and the target table column data is divided into different sub-target table column data;

[0076] According to the mapping relationship, the sub-source table column data and the corresponding sub-target table column data are compared. If the similarity between the two meets the consistency requirement, it is determined that the sub-source table column data and the corresponding sub-target table column data meet the data comparison rule;

[0077] Poll and compare each sub-source table column data with the corresponding sub-target table column data. When any sub-source table column data and the corresponding sub-target table column data meet the data comparison rules, the number of records that meet the data comparison rules is increased by 1 to obtain the total number that meets the data comparison rules.

[0078] For example, the mapping relationship between source table column data and target table column data may be defined as a one-to-one mapping of check fields, the consistency requirement is the same, and the data partition is a partitioning rule that requires partitioning into sub-partitions.

[0079] After obtaining the source table, determine the source table data; after obtaining the target table, determine the target table data; divide the source table and column data into different fields, and divide the target table and column data into different fields; determine the field comparison rule to be greater than, less than, or equal. In this embodiment, equality is used as an example for explanation. The fields in the source table and column and the fields in the target table and column are compared one by one, and the total number of equal fields of the two is calculated, and the total number of equal fields of the two is divided by the total number of source table column data to obtain the data quality accuracy of the comparison checkpoint. The comparison process can be expressed as the following formula:

[0080]

[0081] Among them, Accuracy represents the accuracy of data quality; Count() represents the total number that meets the data comparison rule; Source.fidldN==target.fieldN represents that the Nth source table column data and the target table column data meet the comparison rule; the comparison rule is equal; Count(Source) represents the total number of source table column data.

[0082] It can be understood that by setting the comparison rule as the consistency requirement between the two, the accuracy of the data quality precision calculation is improved.

[0083] Step S33: Based on the data quality accuracy and the pre-set correction threshold, the number of times less than the correction threshold, and the correction retry number threshold, determine whether the first comparison checkpoint meets the correction conditions.

[0084] Step S34, polling and calculating the data quality accuracy of each comparison checkpoint, and determining whether the comparison checkpoint meets the correction conditions; the correction conditions include: the data quality accuracy of the comparison checkpoint is less than the correction threshold, and the number of times the data quality accuracy of the comparison checkpoint is less than the correction threshold is greater than or equal to the correction threshold number threshold; and the number of correction retries for the same recovery checkpoint is less than or equal to the correction retry number threshold.

[0085] In some embodiments, determining whether the comparison checkpoint meets the correction condition may specifically include the following steps:

[0086] Determine whether the data quality accuracy of the comparison checkpoint is less than the deviation correction threshold;

[0087] When the data quality accuracy of the comparison checkpoint is less than the correction threshold, the number of times the data quality accuracy of the comparison checkpoint is less than the correction threshold is increased by 1, and the number of times the data quality accuracy of the comparison checkpoint is less than the correction threshold is updated;

[0088] Determine whether the number of times the data quality accuracy of the updated comparison checkpoint is less than the correction threshold is greater than or equal to a threshold number of times less than the correction threshold;

[0089] If the data quality accuracy of the updated comparison checkpoint is less than the correction threshold for a number of times greater than or equal to the correction threshold, and the number of retries for the same recovery checkpoint is less than or equal to the correction retry threshold, then the comparison checkpoint is determined to meet the correction conditions.

[0090] Specifically, in the process of judging whether the correction conditions are met, the calculated data quality accuracy of the comparison checkpoint can be compared with the correction threshold. If the data quality accuracy is less than the correction threshold, it means that the current data quality comparison result does not meet the expected requirements. The number of times the point is less than the correction threshold is increased by 1, and the data quality accuracy of the comparison checkpoint is calculated again and compared with the correction threshold. When the comparison checkpoint is compared M times and is still less than the correction threshold (M ≥ less than the correction threshold number threshold), it can be determined that the data quality accuracy of the comparison checkpoint does not meet the requirements.

[0091] It is understandable that by comparing the same comparison checkpoint multiple times, we can avoid the problem of excessive changes in source data at a single time point and the lack of synchronization of target data, which leads to too low accuracy of the source and target ends. By comparing multiple times, we can reduce false alarms.

[0092] It is worth noting that if the data quality accuracy of the comparison checkpoint does not meet the requirements, and the number of retries of the same recovery checkpoint is less than or equal to the correction retry threshold, the comparison checkpoint is determined to meet the correction conditions. If the number of retries of the same recovery checkpoint is greater than the correction retry threshold, it means that there are still problems after multiple corrections, and an abnormal alarm is issued to allow users to check and intervene manually.

[0093] Step S35: If the comparison checkpoint meets the correction conditions, the target end restarts the collection thread to collect the source data according to the recovery checkpoint, and uses the recovery checkpoint as the designated checkpoint to cyclically determine the source data and the target data, and the number of correction retries of the recovery checkpoint is increased by 1; wherein the recovery checkpoint is any data point in the comparison checkpoint that does not meet the correction conditions before the current comparison checkpoint.

[0094] When it is determined that the comparison checkpoint meets the correction conditions, the target end is triggered to stop the current data collection according to the recovery checkpoint, and restart the collection thread to re-collect the source data from the recovery checkpoint. Since the recovery checkpoint is any data point in the comparison checkpoint that does not meet the correction conditions before the current comparison checkpoint in the order of data collection (such as the nearest comparison checkpoint before the current comparison checkpoint), the target end can repeat the collection of a section of data again to fill in the abnormal data.

[0095] After the target end has re-collected the data, the data from the source and target ends are collected synchronously and in real time, and data correction and supplementation are performed again to ensure the accuracy of data transmission and the timeliness of supplementation.

[0096] In some embodiments, the method further includes: when the data quality accuracy of the comparison checkpoint is greater than or equal to a correction threshold, obtaining the current offset of the source data; and setting a recovery checkpoint according to the current offset.

[0097] Specifically, when setting the recovery checkpoint, when the data quality accuracy of the comparison checkpoint is greater than or equal to the correction threshold, and when the current data quality comparison result meets expectations, the current offset of the source data can be obtained; according to the current offset, the content corresponding to the current comparison checkpoint is set as the recovery checkpoint; and according to the comparison result of the data quality accuracy of the polled comparison checkpoint, the recovery checkpoint is updated in real time.

[0098] It can be understood that by updating the data quality comparison checkpoint in real time to meet the expected comparison result as the recovery checkpoint, it is ensured that when the data is abnormal, the data can be fully supplemented to avoid the occurrence of incomplete supplement events.

[0099] In some real-time examples, this also includes:

[0100] The target end creates a globally unique index after collecting data from the source end;

[0101] When the target end restarts the collection thread to collect source data according to the recovery checkpoint, it determines whether the collected source data already exists in the target end data based on the global unique index; if it does, the currently collected source data is skipped; if it does not, the currently collected source data is written to the target end, and a global unique index of the currently collected source data is generated.

[0102] Specifically, when the target end repeatedly collects a section of data and performs a fill-in operation on the abnormal data, in order to avoid repeated writing of data, a globally unique index can be used to ensure that the data is not repeated. After writing the data on the target end, a globally unique index can be established based on the hash of the source data. Each time the collection thread is restarted to collect source data again, the data is read from the recovery checkpoint. Before writing the data, it is necessary to determine whether the globally unique index of the data in the task exists. If so, skip the data and continue collecting. If not, write and generate a globally unique index.

[0103] It can be understood that by establishing a global unique index, data non-duplication is guaranteed.

[0104] In some embodiments, the deviation correction threshold includes 95%; the number threshold value less than the deviation correction threshold includes 3 times; and the number threshold value of deviation correction retries includes 9 times.

[0105] It can be understood that by adopting the technical solution provided by this embodiment, during the data transmission process between the source and target ends, the source data and the target data are determined in real time, and the data of the two are compared in real time. The data quality accuracy and the preset correction conditions are used to determine whether correction is needed. When correction is needed, the target end restarts the collection thread to collect the source data according to the set recovery checkpoint, thereby realizing the associated operation of the three processes of data transmission, data consistency check, and abnormal complement. Therefore, by adopting the technical solution of this application, during the data transmission process, it is determined in real time whether the data transmission is abnormal through the correction conditions. When the data transmission is abnormal (that is, the data consistency check fails), the source end collection offset that needs correction is specified, and the data collection thread is restarted to the offset in time to perform abnormal complement. This is convenient and fast, and overcomes the current technical problem of poor timeliness of data correction and complement.

[0106] Based on a general inventive concept, an embodiment of the present invention provides a data de-skew correction and complement device for implementing the above method embodiment.

[0107] Figure 4 A schematic diagram of the structure of a data correction and complement device provided by an embodiment of the present invention, see Figure 4 , the device provided in this embodiment may include the following structure:

[0108] A determination module 41, configured to determine source-end data and target-end data in real time based on a specified checkpoint, wherein the specified checkpoint identifies a starting point for a data consistency check to be performed;

[0109] The calculation module 42 is used to calculate the data quality accuracy of the first comparison checkpoint in the source data and the target data using the initial checkpoint as the first comparison checkpoint; wherein, according to the data collection sequence, the initial checkpoint is any data point after the specified checkpoint;

[0110] The judgment module 43 is used to judge whether the first comparison checkpoint meets the correction condition based on the data quality accuracy and the preset correction threshold, the threshold of the number of times less than the correction threshold, and the threshold of the number of correction retries; poll and calculate the data quality accuracy of each comparison checkpoint, and judge whether the comparison checkpoint meets the correction condition; the correction condition includes: the data quality accuracy of the comparison checkpoint is less than the correction threshold, and the number of times the data quality accuracy of the comparison checkpoint is less than the correction threshold is greater than or equal to the threshold of the number of times less than the correction threshold; and, the number of correction retries of the same recovery checkpoint is less than or equal to the threshold of the number of correction retries;

[0111] The trigger module 44 is used to restart the collection thread on the target end to collect the source end data according to the recovery checkpoint if the comparison checkpoint meets the correction conditions, and use the recovery checkpoint as the designated checkpoint to cyclically determine the source end data and the target end data, and the number of correction retries of the recovery checkpoint is increased by 1; wherein the recovery checkpoint is any data point in the comparison checkpoint that does not meet the correction conditions before the current comparison checkpoint.

[0112] Optionally, a calculation module is specifically used to determine each source table column data of the source end data in the comparison checkpoint, the total number of source table column data, each target table column data of the target end data, and a data comparison rule;

[0113] Calculate the total number of source table column data and target table column data that meet the data comparison rules after verification through the data comparison rules;

[0114] Calculate the ratio of the total number of data that meet the data comparison rules to the total number of source table column data, and use the ratio as the data quality accuracy of the comparison checkpoint.

[0115] Optionally, a calculation module is specifically used to divide the source table column data into different sub-source table column data and divide the target table column data into different sub-target table column data according to the data partition;

[0116] According to the mapping relationship, the sub-source table column data and the corresponding sub-target table column data are compared. If the similarity between the two meets the consistency requirement, it is determined that the sub-source table column data and the corresponding sub-target table column data meet the data comparison rule;

[0117] Poll and compare each sub-source table column data with the corresponding sub-target table column data. When any sub-source table column data and the corresponding sub-target table column data meet the data comparison rules, the number of records that meet the data comparison rules is increased by 1 to obtain the total number that meets the data comparison rules.

[0118] Optionally, the judgment module is further used to obtain the current offset of the source data when the data quality accuracy of the comparison checkpoint is greater than or equal to the correction threshold;

[0119] Set a recovery checkpoint based on the current offset.

[0120] Optionally, a judgment module is specifically used to judge whether the data quality accuracy of the comparison checkpoint is less than the deviation correction threshold;

[0121] When the data quality accuracy of the comparison checkpoint is less than the correction threshold, the number of times the data quality accuracy of the comparison checkpoint is less than the correction threshold is increased by 1, and the number of times the data quality accuracy of the comparison checkpoint is less than the correction threshold is updated;

[0122] Determine whether the number of times the data quality accuracy of the updated comparison checkpoint is less than the correction threshold is greater than or equal to a threshold number of times less than the correction threshold;

[0123] If the data quality accuracy of the updated comparison checkpoint is less than the correction threshold for a number of times greater than or equal to the correction threshold, and the number of retries for the same recovery checkpoint is less than or equal to the correction retry threshold, then the comparison checkpoint is determined to meet the correction conditions.

[0124] Optionally, it also includes: a target-side index module, which is used for the target side to establish a globally unique index after collecting data from the source side;

[0125] When the target end restarts the collection thread to collect source data according to the recovery checkpoint, it determines whether the collected source data already exists in the target end data based on the global unique index; if it does, the currently collected source data is skipped; if it does not, the currently collected source data is written to the target end, and a global unique index of the currently collected source data is generated.

[0126] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0127] It can be understood that by adopting the technical solution provided by this embodiment, during the data transmission process between the source and target ends, the source data and the target data are determined in real time, and the data of the two are compared in real time. The data quality accuracy and the preset correction conditions are used to determine whether correction is needed. When correction is needed, the target end restarts the collection thread to collect the source data according to the set recovery checkpoint, thereby realizing the associated operation of the three processes of data transmission, data consistency check, and abnormal complement. Therefore, by adopting the technical solution of this application, during the data transmission process, it is determined in real time whether the data transmission is abnormal through the correction conditions. When the data transmission is abnormal (that is, the data consistency check fails), the source end collection offset that needs correction is specified, and the data collection thread is restarted to the offset in time to perform abnormal complement. This is convenient and fast, and overcomes the current technical problem of poor timeliness of data correction and complement.

[0128] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0129] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0130] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0131] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0132] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0133] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0134] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0135] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0136] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A data correction and complement method, characterized in that: include: Determining source-end data and target-end data in real time based on a specified checkpoint, wherein the specified checkpoint identifies a starting point for a data consistency check to be performed; Taking the initial checkpoint as the first comparison checkpoint, the data quality accuracy of the first comparison checkpoint in the source data and the target data is calculated; wherein, according to the data collection sequence, the initial checkpoint is any data point after the designated checkpoint; Based on the data quality accuracy and a preset correction threshold, a threshold of times less than the correction threshold, and a threshold of times of correction retries, determining whether the first comparison checkpoint meets the correction condition; Polling and calculating the data quality accuracy of each comparison checkpoint, and determining whether the comparison checkpoint meets the correction condition; the correction condition includes: the data quality accuracy of the comparison checkpoint is less than the correction threshold, and the number of times the data quality accuracy of the comparison checkpoint is less than the correction threshold is greater than or equal to the number of times less than the correction threshold; and the number of correction retries for the same recovery checkpoint is less than or equal to the correction retry number threshold; If the comparison checkpoint meets the correction condition, the target end is made to restart the collection thread to collect source end data according to the recovery checkpoint, and the source end data and the target end data are cyclically determined with the recovery checkpoint as the designated checkpoint, and the correction retry number of the recovery checkpoint is increased by 1; wherein the recovery checkpoint is any data point in the comparison checkpoint that does not meet the correction condition before the current comparison checkpoint; The method for calculating the data quality accuracy of any comparison checkpoint in calculating the data quality accuracy of each comparison checkpoint includes: Determine each source table column data of the source end data, the total number of source table column data, each target table column data of the target end data, and data comparison rules in the comparison checkpoint; Calculating the total number of the source table column data and the target table column data that meet the data comparison rule after verification through the data comparison rule; Calculating the ratio of the total number of the data that meet the data comparison rule to the total number of the source table column data, and using the ratio as the data quality accuracy of the comparison checkpoint; The data comparison rules include: mapping relationship between source table column data and target table column data, data partitioning and consistency requirements; The calculating the total number of the source table column data and the target table column data that meet the data comparison rule after verification through the data comparison rule includes: According to the data partition, the source table column data is divided into different sub-source table column data, and the target table column data is divided into different sub-target table column data; According to the mapping relationship, the sub-source table column data and the corresponding sub-target table column data are compared, and if the degree of similarity between the two meets the consistency requirement, it is determined that the sub-source table column data and the corresponding sub-target table column data meet the data comparison rule; Poll and compare each of the sub-source table column data and the corresponding sub-target table column data. When any sub-source table column data and the corresponding sub-target table column data meet the data comparison rule, the number of records that meet the data comparison rule is incremented by 1 to obtain the total number that meets the data comparison rule.

2. The method according to claim 1, characterized in that Also includes: When the data quality accuracy of the comparison checkpoint is greater than or equal to the correction threshold, obtaining the current offset of the source data; The recovery checkpoint is set according to the current offset.

3. The method according to claim 1, characterized in that The determining whether the comparison checkpoint meets the deviation correction condition includes: Determine whether the data quality accuracy of the comparison checkpoint is less than the deviation correction threshold; When the data quality accuracy of the comparison checkpoint is less than the correction threshold, the number of times the data quality accuracy of the comparison checkpoint is less than the correction threshold is increased by 1, and the number of times the data quality accuracy of the comparison checkpoint is less than the correction threshold is updated; Determine whether the number of times the data quality accuracy of the updated comparison checkpoint is less than the correction threshold is greater than or equal to the threshold number of times less than the correction threshold; If the number of times the data quality accuracy of the updated comparison checkpoint is less than the correction threshold is greater than or equal to the threshold number of times less than the correction threshold, and the number of retries of the same recovery checkpoint is less than or equal to the correction retry threshold, then it is determined that the comparison checkpoint meets the correction condition.

4. The method according to claim 1, characterized in that Also includes: The target end establishes a globally unique index after collecting data from the source end; When the target end restarts the collection thread to collect source end data according to the recovery checkpoint, it is determined whether the collected source end data already exists in the target end data according to the global unique index; If it exists, skip the currently collected source data; If it does not exist, the currently collected source data is written to the target end, and a globally unique index of the currently collected source data is generated.

5. The method according to any one of claims 1 to 4, characterized in that: The deviation correction threshold includes 95%; the number threshold less than the deviation correction threshold includes 3 times; and the number threshold of deviation correction retries includes 9 times.

6. A data correction and complement device, characterized in that: include: A determination module, configured to determine source-end data and target-end data in real time based on a specified checkpoint, wherein the specified checkpoint identifies a starting point for a data consistency check to be performed; A calculation module, used to calculate the data quality accuracy of the first comparison checkpoint in the source data and the target data using the initial checkpoint as the first comparison checkpoint; wherein, according to the data collection sequence, the initial checkpoint is any data point after the specified checkpoint; A judgment module is used to judge whether the first comparison checkpoint meets the correction condition based on the data quality accuracy and a preset correction threshold, a threshold value of times less than the correction threshold, and a threshold value of times of correction retries; poll and calculate the data quality accuracy of each comparison checkpoint, and judge whether the comparison checkpoint meets the correction condition; the correction condition includes: the data quality accuracy of the comparison checkpoint is less than the correction threshold, and the number of times the data quality accuracy of the comparison checkpoint is less than the correction threshold is greater than or equal to the threshold value of times less than the correction threshold; and the number of correction retries of the same recovery checkpoint is less than or equal to the threshold value of times of correction retries; wherein the calculation method of the data quality accuracy of any comparison checkpoint in calculating the data quality accuracy of each comparison checkpoint includes: Determine each source table column data of the source end data, the total number of source table column data, each target table column data of the target end data, and data comparison rules in the comparison checkpoint; Calculating the total number of the source table column data and the target table column data that meet the data comparison rule after verification through the data comparison rule; Calculating the ratio of the total number of the data that meet the data comparison rule to the total number of the source table column data, and using the ratio as the data quality accuracy of the comparison checkpoint; The data comparison rules include: mapping relationship between source table column data and target table column data, data partitioning and consistency requirements; The calculating the total number of the source table column data and the target table column data that meet the data comparison rule after verification through the data comparison rule includes: According to the data partition, the source table column data is divided into different sub-source table column data, and the target table column data is divided into different sub-target table column data; According to the mapping relationship, the sub-source table column data and the corresponding sub-target table column data are compared, and if the degree of similarity between the two meets the consistency requirement, it is determined that the sub-source table column data and the corresponding sub-target table column data meet the data comparison rule; Poll and compare each of the sub-source table column data and the corresponding sub-target table column data, and when any of the sub-source table column data and the corresponding sub-target table column data meet the data comparison rule, the number of records meeting the data comparison rule is incremented by 1 to obtain the total number of records meeting the data comparison rule; A trigger module is used to enable the target end to restart the collection thread to collect source end data according to the recovery checkpoint if the comparison checkpoint meets the correction condition, and to cyclically determine the source end data and the target end data with the recovery checkpoint as the designated checkpoint, and the number of correction retries of the recovery checkpoint is increased by 1; wherein, the recovery checkpoint is any data point in the comparison checkpoint that does not meet the correction condition before the current comparison checkpoint.

7. A data correction and complement device, characterized in that: The invention comprises a processor and a memory, wherein the processor is connected to the memory: Wherein, the processor is used to call and execute the program stored in the memory; The memory is used to store the program, and the program is at least used to execute the data correction and complement method described in any one of claims 1-5.

8. A data correction and complement system, characterized in that: include: A source end, a target end and the data correction and complement device as claimed in claim 7; the source end is used to provide source end data; The target end collects source data from the source end, and when the comparison checkpoint meets the correction condition, restarts the collection thread to collect source data according to the recovery checkpoint; the data correction and complement device is used to perform correction and complement on the target end.

Images