An efficient data processing system and method based on multi-source data comparison

Abstract

The present invention relates to the field of data processing technology, and specifically to an efficient data processing system and method based on multi-source data comparison, in which a self-checking associated data fusion analysis module performs self-checking on the configuration file of each data comparison party based on the data table configuration area, generates a data comparison self-checking associated data project set, and performs fusion determination processing on the elements in the data comparison self-checking associated data project set. The present invention supports flexible configuration of multiple association conditions and data formats to meet the comparison requirements in different scenarios; at the same time, the present invention realizes self-checking of the configuration file of the data comparison party during the comparison process, reducing the impact of interference data in the configuration file on subsequent comparison operations; and introduces fusion determination processing operations on data at different stages in the comparison process, realizing dynamic optimization and screening of data during the comparison process, and realizing effective management of comparison data.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an efficient data processing system and method based on multi-source data comparison. Background Art

[0002] In the current information age, the amount of data generated by various information systems is growing explosively. Especially in the fields of enterprise management, financial analysis, scientific research data, etc., it is often necessary to compare and integrate data from different data sources, different formats, and even redundant and erroneous data. The traditional manual comparison method is not only time-consuming and labor-intensive, but also prone to errors and cannot meet the needs of large-scale data processing.

[0003] Although there are some data comparison tools on the market, most of them have the following shortcomings:

[0004] ① It only supports the comparison of a single or a small number of data tables and cannot efficiently handle the comparison of multiple tables;

[0005] ② The comparison algorithm is single and it is difficult to deal with complex data structures and comparison rules;

[0006] ③Lack of intelligent error identification and correction mechanism, requiring human intervention. Summary of the invention

[0007] The object of the present invention is to provide an efficient data processing system and method based on multi-source data comparison to solve the problems raised in the above background technology.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions: an efficient data processing method based on multi-source data comparison, comprising:

[0009] Aggregate the data forms of each data source into a view page, and build configuration files corresponding to each data in the data table configuration area of ​​the view page based on the acquired data forms;

[0010] Based on the data table configuration area, the configuration file of each data comparison partner is self-calibrated to generate a data comparison self-calibration related data item set, and the elements in the data comparison self-calibration related data item set are fused and judged;

[0011] Based on the fusion judgment processing results of the elements in the data comparison self-checking associated data project set, the configuration files corresponding to each data comparison partner in the data table configuration area are updated; the updated configuration files of each data comparison partner are partitioned, and a data comparison associated object set corresponding to each data partition is generated;

[0012] By means of human-computer interaction, the comparison association mapping items between the data partitions corresponding to each data comparison object are constructed in the comparison rule configuration area in the view page, and the comparison association condition sets corresponding to different data comparison objects are generated;

[0013] Based on the comparison association condition set constructed in the comparison rule configuration area and the data comparison association object set corresponding to each data partition, a multi-core processor or a distributed resource server is used to perform parallel comparison on each data partition to be compared;

[0014] The comparison results between the data partitions of each data comparison partner are statistically summarized to generate the comparison results; based on the fusion judgment processing results of the elements in the data comparison self-calibration associated data project set, the generated comparison results are calibrated and updated, and fed back to the result display area in the view page;

[0015] Extract abnormal comparison data from the comparison results in the result display area, generate a data correction plan for the corresponding abnormal comparison data, and feed it back to the error correction area in the view page;

[0016] After the comparison between the data partitions of each data comparison partner is completed and the error correction area in the view page stops updating, the data in the data table configuration area, comparison rule configuration area, result display area and error correction area in the view page are summarized to generate a data comparison report and feedback to the user.

[0017] Preferably, the number of data comparison partners in the data table configuration area is dynamically set according to the comparison business requirements, and the number of data comparison partners is greater than or equal to 2;

[0018] When the data table configuration area in the view page constructs configuration files corresponding to each data comparison party based on the acquired data form, the user moves the acquired data form to the corresponding data comparison party by dragging or clicking, and uses the summary set of all data forms corresponding to each data comparison party as the configuration file of the corresponding data comparison party, wherein the configuration file in each data comparison party includes one or more data forms;

[0019] When generating the data comparison self-checking associated data item set, perform self-checking on each data comparison partner's configuration file respectively, record the nth data comparison partner's configuration file as Mn, and before performing self-checking and data comparison on the data comparison partner's configuration file, perform data formatting on each configuration file, and unify the data formats of the same type in different data forms in the configuration file;

[0020] Extract each group of data in the configuration file after the data format is unified, compare the data association between different groups of data respectively, and bind different groups of data whose corresponding data association is greater than or equal to the preset value, record any group of data in Mn as A, and take the set consisting of all groups of data except A in all binding results containing A as the self-checking related data item set of A, recorded as AZ; AZ is an element in the data comparison self-checking related data item set corresponding to the configuration file to which A belongs;

[0021] The data association between A and any group of data B in Mn except A is denoted as G （A，B） ,

[0022] When Q2 （A，B） =Q4 （A，B） When G （A，B） =0;

[0023] When Q2 （A，B） ≠Q4 （A，B） hour,

[0024] G （A，B） =exp((Q3 （A，B） -Q1 （A，B) ) / (Q3 （A，B） -Q2 （A，B) )-1)*(Q1 （A，B） +Q4 （A，B） ) / Q3 （A，B） ,

[0025] Among them, Q1 （A，B） Indicates the number of types of data in B that are the same as the corresponding types of data in A after keyword conversion by the mapping keyword group configured by the current user;

[0026] Q2 （A，B） Indicates the number of types of data in B that are different from the corresponding types of data in A after keyword conversion using the mapping keyword group configured by the current user and the mapping keyword group configured in the historical data;

[0027] Q3 （A，B） Represents the number of elements in the intersection of B's ​​type set and A's type set; Q4 （A，B） Represents the number of elements in the intersection of the type set of missing data in B and the type set of missing data in A;

[0028] The mapping keyword group includes two different keywords;

[0029] When performing fusion determination processing on the elements in the data comparison self-checking associated data item set corresponding to the same configuration file, all elements in each self-checking associated data item set in the data comparison self-checking associated data item set corresponding to the configuration file whose data association degree with the group data corresponding to the corresponding self-checking associated data item set is equal to 1 are marked.

[0030] When performing the fusion determination processing operation, the marked elements in the self-checking associated data item set corresponding to the corresponding group data are deleted, the self-checking associated data item set corresponding to the data comparison corresponding to the corresponding configuration file is deleted from the deleted element in the self-checking associated data item set, and the binding relationship between the group data corresponding to the deleted self-checking associated data item set and all elements in its corresponding self-checking associated data item set is released;

[0031] Different fusion judgment processing schemes are generated. After each fusion judgment processing scheme is executed, each group of data marked in the elements of the data comparison self-checking associated data item set corresponding to the corresponding configuration file can be eliminated. The fusion judgment processing scheme with the least number of unmarked group data in the elements deleted in the data comparison self-checking associated data item set corresponding to the corresponding configuration file after executing the fusion judgment processing operation is taken as the best fusion judgment processing scheme, and fusion judgment processing is performed on the elements in the data comparison self-checking associated data item set corresponding to the corresponding configuration file according to the best fusion judgment processing scheme.

[0032] Preferably, when the configuration files corresponding to the respective data in the data table configuration area are updated, the configuration files after the fusion determination processing operation is performed are used as the configuration files after the corresponding data in the data table configuration area is updated;

[0033] When performing data partitioning on the configuration files of each updated data comparison partner, the group data corresponding to each non-empty self-checking related data item set and the elements in each non-empty self-checking related data item set in the data comparison self-checking related data item set corresponding to each updated configuration file are sequentially moved to the end of the corresponding updated configuration file to obtain the configuration file to be partitioned of the corresponding configuration file; and the configuration file to be partitioned is partitioned every preset number of partition groups in order from front to back to obtain different data partitions; each updated configuration file of the data comparison partner corresponds to one or more data partitions;

[0034] The data comparison associated object set corresponding to each data partition includes a first comparison associated object group and a second comparison associated object group of the corresponding data partition.

[0035] The first comparison association object group is composed of all data partitions corresponding to the configuration file to which the corresponding data partition belongs, except for the corresponding data partition;

[0036] The second comparison association object group is composed of data partitions of each configuration file corresponding to the data comparison partner of the configuration file to which the corresponding data partition belongs and the remaining data comparison partners to be compared.

[0037] Preferably, the comparison association mapping item represents a comparison association type pair consisting of various types to be compared in the data partition corresponding to each data comparison object to be compared;

[0038] Each element in the comparison association condition set corresponding to different data comparison objects corresponds to a comparison association condition; the comparison association condition consists of a comparison association type pair, and additional processing operations for corresponding types of data in the corresponding comparison association type pair during comparison.

[0039] Preferably, the comparison result includes a normal comparison data set and an abnormal comparison data set corresponding to each data comparison object.

[0040] The normal comparison data set is a comparison association condition set constructed based on the comparison rule configuration area, and the data comparison association object set corresponding to the corresponding data partition contains group data with elements successfully compared therewith;

[0041] The abnormal comparison data set is a comparison association condition set constructed based on the comparison rule configuration area, and there is no group data with which the elements are successfully compared in the data comparison association object set corresponding to the corresponding data partition;

[0042] Obtaining the group data corresponding to each non-empty self-checking related data item set in the fusion determination processing result of the data comparison self-checking related data item set;

[0043] Mark the group data that belongs to the self-calibration related data item set that is not an empty set and belongs to the normal comparison data set in the comparison result corresponding to each data comparison partner;

[0044] The comparison result of the data after deleting the unlabeled group data belonging to the self-calibration related data item set which is not an empty set and compared with the other party is used as the calibration update result of the comparison result of the corresponding data with the other party.

[0045] Preferably, each element in the abnormal comparison data set in the comparison result corresponds to an abnormal comparison data;

[0046] The data correction scheme for generating corresponding abnormal comparison data comprises the following steps:

[0047] S1. Use preset statistical analysis methods to identify abnormal values ​​or erroneous data in the abnormal comparison data in the comparison results, or perform rule matching on the abnormal comparison data according to preset business rules and data rules to identify non-compliant data;

[0048] S2. For non-compliant data, the preset business rules and data rules automatically correct the abnormal comparison data;

[0049] The abnormal comparison data with abnormal values ​​or erroneous data are compared and fed back to the user to guide the user to make manual corrections; and a data correction plan for the abnormal comparison data in each comparison result is obtained.

[0050] An efficient data processing system based on multi-source data comparison, including: comparison object configuration file construction module, self-checking related data fusion analysis module, data partition management module, related mapping interaction construction module, comparison mode management module, comparison result calibration feedback module, comparison data anomaly correction module and comparison report feedback module,

[0051] The comparison partner configuration file construction module aggregates the data forms of each data source into a view page, and constructs configuration files corresponding to each data comparison partner based on the acquired data forms in the data table configuration area of ​​the view page;

[0052] The self-checking related data fusion analysis module performs self-checking on the configuration files of each data comparison partner based on the data table configuration area, generates a data comparison self-checking related data item set, and performs fusion determination processing on the elements in the data comparison self-checking related data item set;

[0053] The data partition management module updates the configuration files corresponding to each data comparison party in the data table configuration area based on the fusion judgment processing results of the elements in the data comparison self-checking associated data project set; partitions the updated configuration files of each data comparison party, and generates a data comparison associated object set corresponding to each data partition;

[0054] The association mapping interaction construction module constructs the comparison association mapping items between the data partitions corresponding to each data comparison object in the comparison rule configuration area in the view page through human-computer interaction, and generates the comparison association condition sets corresponding to different data comparison objects;

[0055] The comparison mode management module uses a multi-core processor or a distributed resource server to perform parallel comparison on each data partition to be compared based on the comparison association condition set constructed in the comparison rule configuration area and the data comparison association object set corresponding to each data partition;

[0056] The comparison result calibration feedback module statistically summarizes the comparison results between the data partitions of each data comparison partner to generate a comparison result; based on the fusion judgment processing results of the elements in the data comparison self-calibration associated data project set, calibrates and updates the generated comparison result, and feeds back to the result display area in the view page;

[0057] The comparison data anomaly correction module extracts abnormal comparison data from the comparison results in the result display area, generates a data correction scheme for the corresponding abnormal comparison data, and feeds it back to the error correction area in the view page;

[0058] After the comparison between the data partitions of each data comparison partner is completed and the error correction area in the view page stops updating, the comparison report feedback module summarizes the data in the data table configuration area, comparison rule configuration area, result display area and error correction area in the view page, and generates a data comparison report to feedback to the user.

[0059] Preferably, the data partition management module includes a comparison configuration file updating unit and a data comparison associated object analyzing unit.

[0060] The comparison configuration file updating unit updates the configuration files corresponding to the respective data comparison partners in the data table configuration area based on the fusion determination processing result of the elements in the data comparison self-checking related data item set;

[0061] The data comparison associated object analysis unit partitions the updated configuration files of each data comparison partner and generates a data comparison associated object set corresponding to each data partition.

[0062] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0063] (1) The present invention uses distributed computing, parallel processing and other technologies to greatly improve data processing speed and efficiency;

[0064] (2) The present invention combines intelligent error recognition and correction mechanisms to reduce errors and omissions in the comparison process;

[0065] (3) The present invention supports flexible configuration of multiple association conditions and data formats to meet the comparison requirements in different scenarios;

[0066] (4) The present invention provides an intuitive and easy-to-use visual operation interface, reducing the user's learning cost and difficulty of use;

[0067] (5) The present invention realizes self-calibration of the configuration file of the data comparison party during the comparison process, thereby reducing the impact of interference data in the configuration file on subsequent comparison operations; and introduces data fusion judgment processing operations at different stages of the comparison process, thereby realizing dynamic optimization and screening of data during the comparison process, and realizing effective management of comparison data. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0069] Figure 1 It is a structural schematic diagram of an efficient data processing system based on multi-source data comparison of the present invention;

[0070] Figure 2 It is a flow chart of an efficient data processing method based on multi-source data comparison of the present invention. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] See also Figure 1 The present invention provides a technical solution: an efficient data processing system based on multi-source data comparison, including: a comparison object configuration file construction module, a self-checking associated data fusion analysis module, a data partition management module, an associated mapping interactive construction module, a comparison mode management module, a comparison result calibration feedback module, a comparison data anomaly correction module and a comparison report feedback module,

[0073] The comparison partner configuration file construction module aggregates the data forms of each data source into a view page, and constructs configuration files corresponding to each data comparison partner based on the acquired data forms in the data table configuration area of ​​the view page;

[0074] The self-checking related data fusion analysis module performs self-checking on the configuration files of each data comparison partner based on the data table configuration area, generates a data comparison self-checking related data item set, and performs fusion determination processing on the elements in the data comparison self-checking related data item set;

[0075] The data partition management module includes a comparison configuration file update unit and a data comparison associated object analysis unit.

[0076] The comparison configuration file updating unit updates the configuration files corresponding to the respective data comparison partners in the data table configuration area based on the fusion determination processing result of the elements in the data comparison self-checking related data item set;

[0077] The data comparison associated object analysis unit partitions the updated configuration files of each data comparison partner and generates a data comparison associated object set corresponding to each data partition;

[0078] The association mapping interaction construction module constructs the comparison association mapping items between the data partitions corresponding to each data comparison object in the comparison rule configuration area in the view page through human-computer interaction, and generates the comparison association condition sets corresponding to different data comparison objects;

[0079] The comparison mode management module uses a multi-core processor or a distributed resource server to perform parallel comparison on each data partition to be compared based on the comparison association condition set constructed in the comparison rule configuration area and the data comparison association object set corresponding to each data partition;

[0080] The comparison result calibration feedback module statistically summarizes the comparison results between the data partitions of each data comparison partner to generate a comparison result; based on the fusion judgment processing results of the elements in the data comparison self-calibration associated data project set, calibrates and updates the generated comparison result, and feeds back to the result display area in the view page;

[0081] The comparison data anomaly correction module extracts abnormal comparison data from the comparison results in the result display area, generates a data correction scheme for the corresponding abnormal comparison data, and feeds it back to the error correction area in the view page;

[0082] After the comparison between the data partitions of each data comparison partner is completed and the error correction area in the view page stops updating, the comparison report feedback module summarizes the data in the data table configuration area, comparison rule configuration area, result display area and error correction area in the view page, and generates a data comparison report to feedback to the user.

[0083] See also Figure 2 , an efficient data processing method based on multi-source data comparison, comprising:

[0084] Aggregate the data forms of each data source into a view page, and build configuration files corresponding to each data in the data table configuration area of ​​the view page based on the acquired data forms;

[0085] In this embodiment, data sources include databases, files, Web services, etc.;

[0086] The number of data comparison partners in the data table configuration area is dynamically set according to the comparison business requirements, and the number of data comparison partners is greater than or equal to 2;

[0087] When the data table configuration area in the view page constructs configuration files corresponding to each data comparison party based on the acquired data form, the user moves the acquired data form to the corresponding data comparison party by dragging or clicking, and uses the summary set of all data forms corresponding to each data comparison party as the configuration file of the corresponding data comparison party, wherein the configuration file in each data comparison party includes one or more data forms;

[0088] Based on the data table configuration area, the configuration file of each data comparison partner is self-calibrated to generate a data comparison self-calibration related data item set, and the elements in the data comparison self-calibration related data item set are fused and judged;

[0089] When generating a data comparison self-checking associated data item set, each data comparison partner's configuration file is self-checked respectively, and the configuration file of the nth data comparison partner is recorded as Mn. Before executing the self-checking and data comparison of the configuration file of the data comparison partner, each configuration file is formatted, and the same type of data formats in different data forms in the configuration file are unified; in this embodiment, the data format is unified, including: unified data format, such as date format, timestamp format, value format, etc., to ensure that the same type of data in different tables is consistent in format;

[0090] Extract each group of data in the configuration file after the data format is unified, compare the data association between different groups of data respectively, and bind different groups of data whose corresponding data association is greater than or equal to the preset value, record any group of data in Mn as A, and take the set consisting of all groups of data except A in all binding results containing A as the self-checking related data item set of A, recorded as AZ; AZ is an element in the data comparison self-checking related data item set corresponding to the configuration file to which A belongs;

[0091] The data association between A and any group of data B in Mn except A is denoted as G （A，B） ,

[0092] When Q2 （A，B） =Q4 （A，B） When G （A，B） =0;

[0093] When Q2 （A，B） ≠Q4 （A，B） hour,

[0094] G （A，B） =exp((Q3 （A，B） -Q1 （A，B) ) / (Q3 （A，B） -Q2 （A，B) )-1)*(Q1 （A，B） +Q4 （A，B） ) / Q3 （A，B） ,

[0095] Among them, Q1 （A，B） Indicates the number of types of data in B that are the same as the corresponding types of data in A after keyword conversion by the mapping keyword group configured by the current user;

[0096] Q2 （A，B）Indicates the number of types of data in B that are different from the corresponding types of data in A after keyword conversion using the mapping keyword group configured by the current user and the mapping keyword group configured in the historical data;

[0097] Q3 （A，B） Represents the number of elements in the intersection of B's ​​type set and A's type set; Q4 （A，B） Represents the number of elements in the intersection of the type set of missing data in B and the type set of missing data in A;

[0098] The mapping keyword group includes two different keywords;

[0099] When performing fusion determination processing on the elements in the data comparison self-checking associated data item set corresponding to the same configuration file, all elements in each self-checking associated data item set in the data comparison self-checking associated data item set corresponding to the configuration file whose data association degree with the group data corresponding to the corresponding self-checking associated data item set is equal to 1 are marked.

[0100] When performing the fusion determination processing operation, the marked elements in the self-checking associated data item set corresponding to the corresponding group data are deleted, the self-checking associated data item set corresponding to the data comparison corresponding to the corresponding configuration file is deleted from the deleted element in the self-checking associated data item set, and the binding relationship between the group data corresponding to the deleted self-checking associated data item set and all elements in its corresponding self-checking associated data item set is released;

[0101] Different fusion judgment processing schemes are generated. After each fusion judgment processing scheme is executed, each group of data marked in the elements of the data comparison self-checking associated data item set corresponding to the corresponding configuration file can be eliminated. The fusion judgment processing scheme with the least number of unmarked group data in the elements deleted in the data comparison self-checking associated data item set corresponding to the corresponding configuration file after executing the fusion judgment processing operation is taken as the best fusion judgment processing scheme, and fusion judgment processing is performed on the elements in the data comparison self-checking associated data item set corresponding to the corresponding configuration file according to the best fusion judgment processing scheme.

[0102] Based on the fusion judgment processing results of the elements in the data comparison self-checking associated data project set, the configuration files corresponding to each data comparison partner in the data table configuration area are updated; the updated configuration files of each data comparison partner are partitioned, and a data comparison associated object set corresponding to each data partition is generated;

[0103] When updating the configuration files corresponding to the data in the data table configuration area, the configuration files after the fusion determination processing operation is performed are used as the configuration files after the corresponding data in the data table configuration area is updated;

[0104] When performing data partitioning on the configuration files of each updated data comparison partner, the group data corresponding to each non-empty self-checking related data item set and the elements in each non-empty self-checking related data item set in the data comparison self-checking related data item set corresponding to each updated configuration file are sequentially moved to the end of the corresponding updated configuration file to obtain the configuration file to be partitioned of the corresponding configuration file; and the configuration file to be partitioned is partitioned every preset number of partition groups in order from front to back to obtain different data partitions; each updated configuration file of the data comparison partner corresponds to one or more data partitions;

[0105] The data comparison associated object set corresponding to each data partition includes a first comparison associated object group and a second comparison associated object group of the corresponding data partition.

[0106] The first comparison association object group is composed of all data partitions corresponding to the configuration file to which the corresponding data partition belongs, except for the corresponding data partition;

[0107] The second comparison association object group is composed of data partitions of each configuration file corresponding to the data comparison partner of the configuration file to which the corresponding data partition belongs and the remaining data comparison partners to be compared.

[0108] By means of human-computer interaction, the comparison association mapping items between the data partitions corresponding to each data comparison object are constructed in the comparison rule configuration area in the view page, and the comparison association condition sets corresponding to different data comparison objects are generated;

[0109] The comparison association mapping item represents a comparison association type pair consisting of various types to be compared in the data partition corresponding to each data comparison object to be compared;

[0110] Each element in the comparison association condition set corresponding to different data comparison parties corresponds to a comparison association condition; the comparison association condition consists of a comparison association type pair (such as primary key-foreign key relationship, custom field matching, etc.), and additional processing operations on the corresponding type of data in the corresponding comparison association type pair during comparison.

[0111] In this embodiment, the additional processing operation includes a keyword extraction operation and an additional operation on the data (for example, adding a fixed value to one type of data and then comparing it with the other types of data);

[0112] The comparison process in this embodiment includes:

[0113] ① Initialize the comparison result set: create one or more result sets to store the comparison results and difference information.

[0114] ② Compare one by one: traverse a data table (usually a smaller table is selected as the base table), and for each record in the table, search for matching records in other related tables; for the matching records found, perform field-level comparison and record inconsistent fields; if no matching record is found, record that no match was found.

[0115] Based on the comparison association condition set constructed in the comparison rule configuration area and the data comparison association object set corresponding to each data partition, a multi-core processor or a distributed resource server is used to perform parallel comparison on each data partition to be compared;

[0116] The comparison results between the data partitions of each data comparison partner are statistically summarized to generate the comparison results; based on the fusion judgment processing results of the elements in the data comparison self-calibration associated data project set, the generated comparison results are calibrated and updated, and fed back to the result display area in the view page;

[0117] The comparison results include a normal comparison data set and an abnormal comparison data set corresponding to each data comparison object.

[0118] The normal comparison data set is a comparison association condition set constructed based on the comparison rule configuration area, and the data comparison association object set corresponding to the corresponding data partition contains group data with elements successfully compared therewith;

[0119] The abnormal comparison data set is a comparison association condition set constructed based on the comparison rule configuration area, and there is no group data with which the elements are successfully compared in the data comparison association object set corresponding to the corresponding data partition;

[0120] Obtaining the group data corresponding to each non-empty self-checking related data item set in the fusion determination processing result of the data comparison self-checking related data item set;

[0121] Mark the group data that belongs to the self-calibration related data item set that is not an empty set and belongs to the normal comparison data set in the comparison result corresponding to each data comparison partner;

[0122] The comparison result of the data after deleting the unlabeled group data belonging to the self-calibration related data item set which is not an empty set and compared with the other party is used as the calibration update result of the comparison result of the corresponding data with the other party.

[0123] Extract abnormal comparison data from the comparison results in the result display area, generate a data correction plan for the corresponding abnormal comparison data, and feed it back to the error correction area in the view page;

[0124] Each element in the abnormal comparison data set in the comparison result corresponds to an abnormal comparison data;

[0125] In this embodiment, the abnormal comparison data refers to group data with differences during the comparison process, wherein the differences include inconsistent field values ​​in the compared group data, or records in which no matching items are found.

[0126] The data correction scheme for generating corresponding abnormal comparison data comprises the following steps:

[0127] S1. Use preset statistical analysis methods (such as mean, standard deviation, outlier detection, etc.) to identify abnormal values ​​or erroneous data in the comparison results, or match the abnormal comparison data according to preset business rules and data rules to identify non-compliant data;

[0128] S2. For non-compliant data, the preset business rules and data rules automatically correct the abnormal comparison data;

[0129] The abnormal comparison data with abnormal values ​​or erroneous data are compared and fed back to the user to guide the user to make manual corrections; and a data correction plan for the abnormal comparison data in each comparison result is obtained.

[0130] After the comparison between the data partitions of each data comparison partner is completed and the error correction area in the view page stops updating, the data in the data table configuration area, comparison rule configuration area, result display area and error correction area in the view page are summarized to generate a data comparison report and feedback to the user.

[0131] In this embodiment, there are four blocks in the view page, namely the data table configuration area, the comparison rule configuration area, the result display area and the error correction area; the view page uses charts, tables, highlighting and other methods to intuitively display the comparison results and error data.

[0132] When performing interactive functions in the four blocks of the view page, the corresponding blocks provide interactive methods such as dragging and clicking, allowing users to easily configure comparison rules and view comparison results; and support users to view, select and correct erroneous data through the interface;

[0133] The data comparison report generated in this embodiment allows the user to export the comparison results and error data into files in Excel, PDF and other formats as needed to facilitate subsequent analysis and processing.

[0134] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0135] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An efficient data processing method based on multi-source data comparison, characterized in that: include: Aggregate the data forms of each data source into a view page, and build configuration files corresponding to each data in the data table configuration area of ​​the view page based on the acquired data forms; Based on the data table configuration area, the configuration file of each data comparison partner is self-calibrated to generate a data comparison self-calibration related data item set, and the elements in the data comparison self-calibration related data item set are fused and judged; Based on the fusion judgment processing results of the data comparison self-checking related data project set elements, the configuration files corresponding to each data comparison partner in the data table configuration area are updated; Perform data partitioning on the updated configuration files of each data comparison partner, and generate a data comparison associated object set corresponding to each data partition; By means of human-computer interaction, the comparison association mapping items between the data partitions corresponding to each data comparison object are constructed in the comparison rule configuration area in the view page, and the comparison association condition sets corresponding to different data comparison objects are generated; Based on the comparison association condition set constructed in the comparison rule configuration area and the data comparison association object set corresponding to each data partition, a multi-core processor or a distributed resource server is used to perform parallel comparison on each data partition to be compared; Comparison results between data partitions of each data comparison object are statistically summarized to generate comparison results; Based on the fusion judgment processing results of the elements in the data comparison self-calibration associated data project set, the generated comparison results are calibrated and updated, and fed back to the result display area in the view page; Extract abnormal comparison data from the comparison results in the result display area, generate a data correction plan for the corresponding abnormal comparison data, and feed it back to the error correction area in the view page; After the comparison between the data partitions of each data comparison partner is completed and the error correction area in the view page stops updating, the data in the data table configuration area, comparison rule configuration area, result display area and error correction area in the view page are summarized to generate a data comparison report and feedback to the user.

2. The efficient data processing method based on multi-source data comparison according to claim 1, characterized in that: The number of data comparison partners in the data table configuration area is dynamically set according to the comparison business requirements, and the number of data comparison partners is greater than or equal to 2; When the data table configuration area in the view page constructs configuration files corresponding to each data comparison party based on the acquired data form, the user moves the acquired data form to the corresponding data comparison party by dragging or clicking, and uses the summary set of all data forms corresponding to each data comparison party as the configuration file of the corresponding data comparison party, wherein the configuration file in each data comparison party includes one or more data forms; When generating the data comparison self-checking associated data item set, perform self-checking on each data comparison partner's configuration file respectively, record the nth data comparison partner's configuration file as Mn, and before performing self-checking and data comparison on the data comparison partner's configuration file, perform data formatting on each configuration file, and unify the data formats of the same type in different data forms in the configuration file; Extract each group of data in the configuration file after the data format is unified, compare the data association between different groups of data respectively, and bind different groups of data whose corresponding data association is greater than or equal to the preset value, record any group of data in Mn as A, and take the set consisting of all groups of data except A in all binding results containing A as the self-checking related data item set of A, recorded as AZ; AZ is an element in the data comparison self-checking related data item set corresponding to the configuration file to which A belongs; The data association between A and any group of data B in Mn except A is denoted as G （A，B） , When Q2 （A，B） =Q4 （A，B） When G （A，B） =0; When Q2 （A，B） ≠Q4 （A，B） hour, <h2 style=";text-align:left;direction:ltr">G<h2 style=";text-align:left;direction:ltr"> （A，B） <h2 style=";text-align:left;direction:ltr"> =exp((Q3<h2 style=";text-align:left;direction:ltr"> （A，B） <h2 style=";text-align:left;direction:ltr"> -Q1<h2 style=";text-align:left;direction:ltr"> （A，B) <h2 style=";text-align:left;direction:ltr"> ) / (Q3<h2 style=";text-align:left;direction:ltr"> （A，B） <h2 style=";text-align:left;direction:ltr"> -Q2<h2 style=";text-align:left;direction:ltr"> （A，B) <h2 style=";text-align:left;direction:ltr"> )-1)*(Q1<h2 style=";text-align:left;direction:ltr"> （A，B） <h2 style=";text-align:left;direction:ltr"> +Q4<h2 style=";text-align:left;direction:ltr"> （A，B） <h2 style=";text-align:left;direction:ltr"> ) / Q3<h2 style=";text-align:left;direction:ltr"> （A，B） <h2 style=";text-align:left;direction:ltr"> , Among them, Q1 （A，B） Indicates the number of types of data in B that are the same as the corresponding types of data in A after keyword conversion by the mapping keyword group configured by the current user; Q2 （A，B） Indicates the number of types of data in B that are different from the corresponding types of data in A after keyword conversion using the mapping keyword group configured by the current user and the mapping keyword group configured in the historical data; Q3 （A，B） Represents the number of elements in the intersection of B's ​​type set and A's type set; Q4 （A，B） Represents the number of elements in the intersection of the type set of missing data in B and the type set of missing data in A; The mapping keyword group includes two different keywords; When performing fusion determination processing on the elements in the data comparison self-checking associated data item set corresponding to the same configuration file, all elements in each self-checking associated data item set in the data comparison self-checking associated data item set corresponding to the configuration file whose data association degree with the group data corresponding to the corresponding self-checking associated data item set is equal to 1 are marked. When performing the fusion determination processing operation, the marked elements in the self-checking associated data item set corresponding to the corresponding group data are deleted, the self-checking associated data item set corresponding to the data comparison corresponding to the corresponding configuration file is deleted from the deleted element in the self-checking associated data item set, and the binding relationship between the group data corresponding to the deleted self-checking associated data item set and all elements in its corresponding self-checking associated data item set is released; Different fusion judgment processing schemes are generated. After each fusion judgment processing scheme is executed, each group of data marked in the elements of the data comparison self-checking associated data item set corresponding to the corresponding configuration file can be eliminated. The fusion judgment processing scheme with the least number of unmarked group data in the elements deleted in the data comparison self-checking associated data item set corresponding to the corresponding configuration file after executing the fusion judgment processing operation is taken as the best fusion judgment processing scheme, and fusion judgment processing is performed on the elements in the data comparison self-checking associated data item set corresponding to the corresponding configuration file according to the best fusion judgment processing scheme.

3. The efficient data processing method based on multi-source data comparison according to claim 1, characterized in that: When updating the configuration files corresponding to the data in the data table configuration area, the configuration files after the fusion determination processing operation is performed are used as the configuration files after the corresponding data in the data table configuration area is updated; When performing data partitioning on the configuration files of each updated data comparison partner, the group data corresponding to each non-empty self-checking related data item set and the elements in each non-empty self-checking related data item set in the data comparison self-checking related data item set corresponding to each updated configuration file are sequentially moved to the end of the corresponding updated configuration file to obtain the configuration file to be partitioned of the corresponding configuration file; and the configuration file to be partitioned is partitioned every preset number of partition groups in order from front to back to obtain different data partitions; each updated configuration file of the data comparison partner corresponds to one or more data partitions; The data comparison associated object set corresponding to each data partition includes a first comparison associated object group and a second comparison associated object group of the corresponding data partition. The first comparison association object group is composed of all data partitions corresponding to the configuration file to which the corresponding data partition belongs, except for the corresponding data partition; The second comparison association object group is composed of data partitions of each configuration file corresponding to the data comparison partner of the configuration file to which the corresponding data partition belongs and the remaining data comparison partners to be compared.

4. The efficient data processing method based on multi-source data comparison according to claim 2 is characterized in that: The comparison association mapping item represents a comparison association type pair consisting of various types to be compared in the data partition corresponding to each data comparison object to be compared; Each element in the comparison association condition set corresponding to different data comparison objects corresponds to a comparison association condition; the comparison association condition consists of a comparison association type pair, and additional processing operations for corresponding types of data in the corresponding comparison association type pair during comparison.

5. The efficient data processing method based on multi-source data comparison according to claim 4 is characterized in that: The comparison results include a normal comparison data set and an abnormal comparison data set corresponding to each data comparison object. The normal comparison data set is a comparison association condition set constructed based on the comparison rule configuration area, and the data comparison association object set corresponding to the corresponding data partition contains group data with elements successfully compared therewith; The abnormal comparison data set is a comparison association condition set constructed based on the comparison rule configuration area, and there is no group data with which the elements are successfully compared in the data comparison association object set corresponding to the corresponding data partition; Obtaining the group data corresponding to each non-empty self-checking related data item set in the fusion determination processing result of the data comparison self-checking related data item set; Mark the group data that belongs to the self-calibration related data item set that is not an empty set and belongs to the normal comparison data set in the comparison result corresponding to each data comparison partner; The comparison result of the data after deleting the unlabeled group data belonging to the self-calibration related data item set which is not an empty set and compared with the other party is used as the calibration update result of the comparison result of the corresponding data with the other party.

6. The efficient data processing method based on multi-source data comparison according to claim 5 is characterized in that: Each element in the abnormal comparison data set in the comparison result corresponds to an abnormal comparison data; The data correction scheme for generating corresponding abnormal comparison data comprises the following steps: S1. Use preset statistical analysis methods to identify abnormal values ​​or erroneous data in the abnormal comparison data in the comparison results, or perform rule matching on the abnormal comparison data according to preset business rules and data rules to identify non-compliant data; S2. For non-compliant data, the preset business rules and data rules automatically correct the abnormal comparison data; The abnormal comparison data with abnormal values ​​or erroneous data are compared and fed back to the user to guide the user to make manual corrections; and a data correction plan for the abnormal comparison data in each comparison result is obtained.

7. An efficient data processing system based on multi-source data comparison, characterized in that: include: Comparison object configuration file construction module, self-checking related data fusion analysis module, data partition management module, related mapping interaction construction module, comparison method management module, comparison result calibration feedback module, comparison data anomaly correction module and comparison report feedback module, The comparison partner configuration file construction module aggregates the data forms of each data source into a view page, and constructs configuration files corresponding to each data comparison partner based on the acquired data forms in the data table configuration area of ​​the view page; The self-checking related data fusion analysis module performs self-checking on the configuration files of each data comparison partner based on the data table configuration area, generates a data comparison self-checking related data item set, and performs fusion determination processing on the elements in the data comparison self-checking related data item set; The data partition management module updates the configuration files corresponding to the data comparison objects in the data table configuration area based on the fusion judgment processing results of the elements in the data comparison self-checking related data project set; Perform data partitioning on the updated configuration files of each data comparison partner, and generate a data comparison associated object set corresponding to each data partition; The association mapping interaction construction module constructs the comparison association mapping items between the data partitions corresponding to each data comparison object in the comparison rule configuration area in the view page through human-computer interaction, and generates the comparison association condition sets corresponding to different data comparison objects; The comparison mode management module uses a multi-core processor or a distributed resource server to perform parallel comparison on each data partition to be compared based on the comparison association condition set constructed in the comparison rule configuration area and the data comparison association object set corresponding to each data partition; The comparison result calibration feedback module statistically summarizes the comparison results between the data partitions of each data comparison object to generate a comparison result; Based on the fusion judgment processing results of the elements in the data comparison self-calibration associated data project set, the generated comparison results are calibrated and updated, and fed back to the result display area in the view page; The comparison data anomaly correction module extracts abnormal comparison data from the comparison results in the result display area, generates a data correction scheme for the corresponding abnormal comparison data, and feeds it back to the error correction area in the view page; After the comparison between the data partitions of each data comparison partner is completed and the error correction area in the view page stops updating, the comparison report feedback module summarizes the data in the data table configuration area, comparison rule configuration area, result display area and error correction area in the view page, and generates a data comparison report to feedback to the user.

8. The efficient data processing system based on multi-source data comparison according to claim 7, characterized in that: The data partition management module includes a comparison configuration file update unit and a data comparison associated object analysis unit. The comparison configuration file updating unit updates the configuration files corresponding to the respective data comparison partners in the data table configuration area based on the fusion determination processing result of the elements in the data comparison self-checking related data item set; The data comparison associated object analysis unit partitions the updated configuration files of each data comparison partner and generates a data comparison associated object set corresponding to each data partition.

Images