Data comparison method and device

Abstract

The invention discloses a data comparison method and device, and belongs to the technical field of information. The method comprises the steps that a comparison task is created based on data source configuration, data position information is set, and a built-in comparison rule is preset; when the comparison task is executed, the data to be compared are cleaned, node-level refined comparison is carried out by utilizing a comparison strategy, differences are marked, and results are stored; performing multi-dimensional aggregation statistical analysis on the basis of a comparison result, and generating a field consistency rate and difference classification statistics; the device comprises a built-in rule component, an external rule component, a comparison difference marking component and a result statistics component, and supports dynamic addition of external rules for secondary comparison optimization. According to the scheme, the efficiency and accuracy of data consistency verification are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of information technology, and in particular relates to a data comparison method and apparatus. Background Technology

[0002] In the field of information technology, data inconsistency has become a major challenge in system development and maintenance. With increasing system complexity and surging data volumes, especially in scenarios such as database version upgrades, system development framework migrations, data conversions, and system IT innovation transformations, inconsistencies between past and present data often lead to system failures or business interruptions. For example, when migrating data between different database systems, the same table may differ in storage structure or field names; the order of data items in a single record may change due to different requests, making manual comparison extremely complex and error-prone. Furthermore, environmental changes such as timestamps, hardware status, and network communication conditions further increase the difficulty of maintaining data consistency. Traditional manual comparison methods can no longer meet the high standards of data consistency required by modern information systems, necessitating an efficient, accurate, and customizable data comparison solution.

[0003] Currently, various data comparison tools and methods exist in the market. The most common are text-based comparison tools (such as diff tools), which can compare file content line by line and mark differences. Another method is database checksum-based comparison technology, which quickly identifies changes by calculating the hash value of data blocks. In addition, some professional data integration platforms provide data quality inspection functions, including field-level value range verification and format verification. In the patent field, there are already technical solutions similar to "A document comparison method, apparatus, electronic device, chip and medium" (CN118798152A), which uses document numbers to obtain different versions of data for content comparison. These existing technologies alleviate the workload of data comparison to some extent and provide basic support for system development and maintenance.

[0004] While existing technologies offer some solutions, significant shortcomings remain: First, general-purpose comparison tools lack customization capabilities for specific business scenarios and cannot adapt to the unique comparison needs of different data structures. Second, existing methods typically only provide simple difference markers, lacking intelligent classification and statistical analysis functions for difference types, leading to inefficient subsequent problem localization. Third, most solutions do not support dynamic adjustment of comparison rules; when initial comparison results are found to be inconsistent with expectations, the entire comparison process often needs to be re-executed, resulting in wasted resources. Finally, existing technologies have weak capabilities for assessing the overall consistency of batch data, making it difficult to provide the statistical indicators needed for macro-level decision-making. Especially when processing structured data such as JSON and XML, existing tools often cannot delve into the internal structure of the data for fine-grained comparisons, nor can they flexibly ignore non-critical differences.

[0005] Therefore, the current data comparison field urgently needs a new solution that can support custom rules, provide intelligent difference classification, support secondary comparison optimization, and have comprehensive statistical analysis capabilities. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies by providing an efficient, accurate, and customizable data comparison solution. It supports user-defined comparison rules, provides a difference classification and summarization function, and supports secondary comparison optimization.

[0007] This invention is achieved through the following technical solution: A first aspect of the present invention provides a data comparison method, the method comprising: Step S1: Create a comparison task based on the data source configuration; Step S2: Execute the comparison task according to the created comparison task and store the comparison results in the result database; Step S3: Based on the comparison results of the comparison task, perform multidimensional aggregation statistical analysis and summary on the differences, generate field consistency rate data for each group of data participating in the comparison, classify and count data with the same reasons for difference, and also generate field consistency rate data for the overall comparison task. Store the data in the statistical database and return it to the front end.

[0008] Furthermore, in step S1, the creation of the comparison task based on the data source configuration also includes: Configure the data location information for comparison based on the data source settings; Pre-configure and store built-in comparison rules according to requirements.

[0009] The setting of comparison data location information is used to obtain comparison data and detect the data types involved in the comparison based on preset data location information; The built-in comparison rules are set according to the type of data to be compared. Different comparison strategies and methods are used for different data types, including plain text content, XML messages, and JSON messages.

[0010] Furthermore, in step S2, the step of executing the comparison task according to the created comparison task and storing the comparison results in the result library also includes: Step S201: Start the comparison task, pull the data to be compared from the corresponding location according to the data source configuration, and at the same time pull the comparison strategy corresponding to the comparison task from the stored built-in rules; Step S202: Clean the data to be compared according to the preset rules to obtain the cleaned data to be compared, and use the comparison strategy to compare the data to obtain the comparison results, and store the comparison results in the result library.

[0011] Furthermore, in step S202, the cleaning process includes noise removal, character replacement, node extraction, etc. The comparison strategy used to compare the data and obtain the comparison results includes: Step S202-1: Match each node of the expected data with the actual data one by one. Based on the matched nodes, extract the corresponding values ​​from the expected data and the actual data respectively and compare them to determine whether there is a difference. Step S202-2: Mark the differences for the data with comparison discrepancies, record the location and corresponding value of the difference between the expected and actual data, count the number of differences, and store the recorded comparison results and the results for the data without comparison discrepancies in the result database until all data comparisons are completed.

[0012] Furthermore, the difference labeling refers to labeling data with differences based on the location and characteristics of the differences and establishing a correlation with the original data.

[0013] Furthermore, in step S3, the multidimensional aggregation statistical analysis and summarization of the differences based on the comparison results of the comparison task also includes: The comparison results data are retrieved from the comparison result database. The difference data is aggregated, classified and statistically analyzed according to different dimensions. The various indicators of the comparison in the entire task are recorded and the statistical information is stored in the statistical database.

[0014] Furthermore, the different dimensions include the same cause of failure and the same location of difference; The various metrics for the entire task comparison include the total number of fields involved in the comparison, the number of fields that match, the number and details of fields that differ, the number of successful comparisons, and the number of failed comparisons.

[0015] Furthermore, the data comparison method described in the first aspect of the present invention further includes setting external comparison rules in the original preset built-in comparison rules. After the external rules are set, a second comparison is performed based on the unsatisfactory original comparison results, and the comparison differences and comparison results are updated at the same time.

[0016] Furthermore, the secondary comparison specifically includes: Obtain external comparison rules, compare the comparison data again based on the original data according to the rules, and re-mark the differences; The comparison difference data is updated to the comparison result database, and the difference issues are then subjected to multidimensional aggregation statistical analysis and summary.

[0017] Furthermore, the data comparison method described in the first aspect of the present invention also includes a rollback mechanism, which is used to delete the configured external comparison rules and reconfigure them to generate new comparison results when the external comparison rules are incorrectly set or the defects of the original comparison error nodes are resolved through iterative development.

[0018] A second aspect of the present invention provides a data comparison device, the device comprising a built-in rule component, an external rule component, a comparison difference marker component, and a comparison result statistics component, wherein: The built-in rule component is used to set the corresponding comparison strategy and method based on the known data types involved in the comparison; The external rule component is used to add more refined comparison rules after the initial data comparison is completed. The comparison difference labeling component is used to label and classify the differences in the data when the data comparison is inconsistent during the comparison process, and then cluster the same differences. The comparison result statistics component is used to perform individual data statistics and overall data statistics on the data when the data comparison is completed, forming an overall result after the data comparison. The overall data statistics is a summary of the statistical results of individual data.

[0019] Compared with the prior art, the present invention has the following advantages: 1. The present invention has highly flexible rule configuration capabilities, supports user-defined comparison rules, including built-in rule presets and dynamic addition of external rules; allows for secondary comparison optimization based on the initial comparison results, significantly improving comparison accuracy; and provides fine-grained rule setting options such as noise filtering, partial character equivalence, and node ignoring.

[0020] 2. This invention employs a difference labeling component to automatically classify and cluster inconsistent data, providing multidimensional statistical functions and supporting aggregation analysis based on differences by type, location, and other dimensions, facilitating problem localization. Furthermore, this invention reduces the difficulty of interpreting results and the need for manual intervention by automatically generating intuitive visual comparisons, and also supports a rollback mechanism for comparison results, ensuring the controllability of the comparison process.

[0021] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the overall process of a data comparison method according to the present invention; Figure 2 This is a schematic diagram of the comparison task process of a data comparison method according to the present invention; Figure 3 This is a schematic diagram of the comparison result analysis process of a data comparison method according to the present invention; Figure 4 This is a structural diagram of a data comparison device according to the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Example 1: In this embodiment, please refer to Figure 1 A data comparison method is provided, the method comprising the following steps: S1: Create a comparison task based on the data source configuration; Furthermore, in S1, the creation of the comparison task based on the data source configuration also includes: Configure the data location information for comparison based on the data source; pre-set and store the built-in comparison rules as needed.

[0026] Furthermore, the setting of comparison data location information is used to obtain comparison data and detect the data types participating in the comparison based on the preset data location information; the built-in comparison rules are set according to the type of data participating in the comparison, that is, different comparison strategies and methods are used for different data types, wherein the data types include plain text content, XML messages, and JSON messages.

[0027] In one embodiment, taking JSON message data as an example, the built-in comparison rules are introduced. For JSON message data, if no comparison rules are set, the default is recursive depth comparison, that is, starting from the root node, recursively traversing all fields layer by layer, and comparing keys and values ​​at the same time. If comparison rules are set, the data to be compared is processed according to the comparison rules, and then the processed data is compared. The comparison rules for JSON message data may include: only comparing nodes node1 / node2 / node3 in the JSON data, and ignoring nodes ignore1 / ignore2 during the comparison.

[0028] S2: Based on the created comparison task, execute the comparison task and store the comparison results in the result database; Furthermore, in this embodiment, as Figure 2 As shown, in S2, the process of executing the comparison task based on the created comparison task and storing the comparison results in the result library further includes: S201: Start the comparison task, pull the data to be compared from the corresponding location according to the data source configuration, and pull the comparison strategy corresponding to the comparison task from the stored built-in rules; S202: Clean the data to be compared according to the preset rules to obtain the cleaned data to be compared, and use the comparison strategy to compare the data to obtain the comparison results, and store the comparison results in the result library.

[0029] Furthermore, the cleaning process includes noise removal, character replacement, and node extraction; The comparison strategy used to compare the data and obtain the comparison results includes: S202-1: Match each node of the expected data with the actual data one by one, and take the corresponding values ​​from the expected data and the actual data according to the matched nodes to compare and determine whether there is a difference; S202-2: Mark the differences for data with discrepancies, record the location and corresponding value of the differences between the expected and actual data, count the number of differences, and store the recorded comparison results and the results for data without discrepancies in the result database until all data comparisons are completed.

[0030] Furthermore, the comparison difference refers to inconsistent values ​​of corresponding nodes or mismatched node information; The aforementioned difference labeling refers to labeling data with discrepancies based on the location and characteristics of the discrepancies and establishing a correlation with the original data.

[0031] S3: Based on the comparison results of the comparison task, perform multi-dimensional aggregation statistical analysis and summary on the differences, generate field consistency rate data for each group of data participating in the comparison, classify and count data with the same reasons for difference, and also generate field consistency rate data for the overall comparison task. Store the data in the statistical database and return it to the front end.

[0032] Furthermore, in this embodiment, as Figure 3 As shown in S3, the multidimensional aggregation statistical analysis and summary of the difference problem based on the comparison results of the comparison task also includes: searching and obtaining comparison result data from the comparison result library, aggregating and classifying the difference data according to different dimensions, recording various indicators of the entire task comparison in the statistics, and storing the statistical information in the statistical library. Furthermore, the various indicators of the entire task comparison in the recorded statistics include the total number of fields participating in the comparison task, the number of fields that match, the number and details of fields with differences, the number of successful comparisons, and the number of failures; the different dimensions include the same reason for failure, the same location of difference, etc.

[0033] Furthermore, the data comparison method further includes: setting external comparison rules within the original preset built-in comparison rules; after the external rules are set, based on the unsatisfactory original comparison results, applying the external rules to perform a second comparison, and simultaneously updating the comparison differences and comparison results, wherein... The secondary comparison specifically includes: Obtain external comparison rules, compare the comparison data again based on the original data according to the rules, and re-mark the differences; The comparison difference data is updated to the comparison result database, and the difference issues are then subjected to multidimensional aggregation statistical analysis and summary.

[0034] Furthermore, in one embodiment, the data comparison method also includes a comparison result rollback mechanism, which is used to delete the configured external comparison rules and reconfigure them to generate new comparison results when the external comparison rules are incorrectly set or the defects of the original comparison error node are resolved through iterative development.

[0035] Example 2: In this embodiment, as Figure 4 As shown, a data comparison device is provided for performing the aforementioned data comparison method. The device includes a built-in rule component, an external rule component, a comparison difference marker component, and a comparison result statistics component, wherein: The built-in rule component is used to set the corresponding comparison strategy and method based on the known data types involved in the comparison; The external rule component is used to add more refined comparison rules after the initial data comparison is completed. The comparison difference labeling component is used to label and classify the differences in the data when the data comparison is inconsistent during the comparison process, and then cluster the same differences. The comparison result statistics component is used to perform individual data statistics and overall data statistics on the data when the data comparison is completed, forming an overall result after the data comparison. The overall data statistics is a summary of the statistical results of individual data.

[0036] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data comparison method, characterized in that, The method includes: Create a comparison task based on the data source configuration; Based on the created comparison task, execute the comparison task and store the comparison results in the result database; Based on the comparison results of the comparison task, multidimensional aggregation statistical analysis and summarization are performed on the discrepancies. Field consistency rate data is generated for each group of data participating in the comparison. Data with the same reasons for discrepancies are classified and counted. Field consistency rate data is also generated for the entire comparison task. The data is stored in the statistical database and returned to the front end.

2. The data comparison method as described in claim 1, characterized in that, The method of creating a comparison task based on data source configuration also includes: Configure the comparison data location information according to the data source configuration; pre-set and store the built-in comparison rules according to requirements; among them; The setting of comparison data location information is used to obtain comparison data and detect the data types participating in the comparison based on preset data location information; The built-in comparison rules are set according to the type of data to be compared. Different comparison strategies and methods are used for different data types, including plain text content, XML messages, and JSON messages.

3. The data comparison method as described in claim 2, characterized in that, The step of executing the comparison task according to the created comparison task and storing the comparison results in the result library also includes: Start the comparison task, pull the data to be compared from the corresponding location according to the data source configuration, and at the same time pull the comparison strategy corresponding to the comparison task from the stored built-in rules; The data to be compared is cleaned according to preset rules to obtain cleaned data to be compared. The comparison strategy is then used to compare the data to obtain the comparison results, which are then stored in the result database.

4. The data comparison method as described in claim 3, characterized in that: The cleaning process includes noise removal, character replacement, and node extraction.

5. The data comparison method as described in claim 3, characterized in that, The comparison strategy used to compare the data and obtain the comparison results includes: The expected data is matched one by one with each node of the actual data. The matched data is then compared with the corresponding values ​​from the expected data and the actual data according to the matched nodes to determine whether there are any differences. For data with discrepancies, mark the discrepancies, record the location and corresponding value of the discrepancies between the expected and actual data, count the number of discrepancies, and store the recorded comparison results and the results for data without discrepancies in the result database until all data comparisons are completed.

6. The data comparison method as described in claim 5, characterized in that: The difference labeling refers to labeling data with differences based on the location and characteristics of the differences, and establishing a correlation with the original data.

7. The data comparison method as described in claim 1, characterized in that, The comparison results based on the comparison task, which involve multidimensional aggregation statistical analysis and summarization of differences, also include: The comparison results data are retrieved from the comparison result database. The difference data is aggregated, classified and statistically analyzed according to different dimensions. The various indicators of the comparison in the entire task are recorded and the statistical information is stored in the statistical database.

8. The data comparison method as described in claim 7, characterized in that: The different dimensions include the same cause of failure and the same location of difference; The various metrics for the entire task comparison include the total number of fields involved in the comparison, the number of fields that match, the number and details of fields that differ, the number of successful comparisons, and the number of failed comparisons.

9. A data comparison method as described in any one of claims 1 to 8, characterized in that: The data comparison method further includes setting external comparison rules in the original preset built-in comparison rules. After the external rules are set, a second comparison is performed based on the unsatisfactory original comparison results, and the comparison differences and comparison results are updated at the same time.

10. The data comparison method as described in claim 9, characterized in that, The secondary comparison specifically includes: Obtain external comparison rules, compare the comparison data again based on the original data according to the rules, and re-mark the differences; The comparison difference data is updated to the comparison result database, and the difference issues are then subjected to multidimensional aggregation statistical analysis and summary.

11. The data comparison method as described in claim 10, characterized in that: The data comparison method also includes a rollback mechanism, which is used to delete the configured external comparison rules and reconfigure them to generate new comparison results when the external comparison rules are incorrectly set or the defects of the original comparison error nodes are resolved through iterative development.

12. A data comparison apparatus for performing a data comparison method as described in any one of claims 1 to 11, characterized in that, The device includes a built-in rule component, an external rule component, a comparison difference marker component, and a comparison result statistics component, wherein: The built-in rule component is used to set the corresponding comparison strategy and method based on the known data types involved in the comparison; The external rule component is used to add more refined comparison rules after the initial data comparison is completed. The comparison difference labeling component is used to label and classify the differences in the data when the data comparison is inconsistent during the comparison process, and then cluster the same differences. The comparison result statistics component is used to perform individual data statistics and overall data statistics on the data when the data comparison is completed, forming an overall result after the data comparison. The overall data statistics is a summary of the statistical results of individual data.

Images