Heterogeneous data comparison method and device

Abstract

The present invention provides a heterogeneous data comparison method and device, which belongs to the field of financial technology or other technical fields. The heterogeneous data comparison method includes: obtaining two data tables according to a data comparison instruction, summing and comparing the records in each column of the two data tables according to the data type of the records in each column of the data table; determining the data comparison strategy according to the comparison result of the sum comparison, the unique identifier of the data table and the data volume; splicing the records in the data table into corresponding target strings according to the data comparison strategy, and comparing the target strings of the two data tables to generate data table comparison results. The present invention can quickly verify the data table after the migration or synchronization of heterogeneous data sources, solve the performance bottleneck, and improve the comparison efficiency.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a heterogeneous data comparison method and device. Background Art

[0002] Comparing data in the same table of heterogeneous data sources is a common problem. The differences in table definitions and the consistency of table data contents need to be fully verified. If you compare them manually one by one, it is easy to miss something and it is difficult to eliminate some detailed differences.

[0003] Currently, some software can support table comparison of heterogeneous databases, such as tablediff, sqldelta and other database management related tools. However, the current software is very likely to have memory overflow and software crash when comparing large data tables. Summary of the invention

[0004] The main purpose of the embodiments of the present invention is to provide a heterogeneous data comparison method and device to quickly verify the data table after the heterogeneous data source is migrated or synchronized, solve the performance bottleneck, and improve the comparison efficiency.

[0005] In order to achieve the above object, an embodiment of the present invention provides a heterogeneous data comparison method, comprising:

[0006] Obtain two data tables according to the data comparison instruction, and sum and compare the records in each column of the two data tables according to the data types of the records in each column of the data tables;

[0007] Determine the data comparison strategy according to the comparison result of the sum comparison, the unique identifier of the data table and the data volume;

[0008] According to the data comparison strategy, each record of the data table is spliced ​​into a corresponding target string, and the target strings of the two data tables are compared to generate a data table comparison result.

[0009] The embodiment of the present invention further provides a heterogeneous data comparison device, comprising:

[0010] A summing and comparing module is used to obtain two data tables according to a data comparison instruction, and to sum and compare the records in each column of the two data tables according to the data types of the records in each column of the data tables;

[0011] A comparison strategy determination module, used to determine a data comparison strategy according to the comparison result of the sum comparison, the unique identifier of the data table and the data volume;

[0012] The comparison result generation module is used to splice each record of the data table into a corresponding target string according to the data comparison strategy, and compare the target strings of the two data tables to generate a data table comparison result.

[0013] An embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the steps of the heterogeneous data comparison method when executing the computer program.

[0014] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the heterogeneous data comparison method are implemented.

[0015] The heterogeneous data comparison method and device of the embodiment of the present invention first sum and compare the records of each column of the two data tables according to the data type, and then determine the data comparison strategy according to the comparison result of the sum and comparison to splice the records of the data tables into corresponding target strings, compare the target strings of the two data tables to generate data table comparison results, and can quickly verify the data tables after the migration or synchronization of heterogeneous data sources, solve performance bottlenecks, and improve comparison efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is a flow chart of a heterogeneous data comparison method according to an embodiment of the present invention;

[0018] Figure 2 is a flow chart of a heterogeneous data comparison method in another embodiment of the present invention;

[0019] Figure 3 is a flow chart of a heterogeneous data comparison method in another embodiment of the present invention;

[0020] Figure 4 is a flow chart of concatenating target character strings in one embodiment of the present invention;

[0021] Figure 5 is a structural block diagram of a heterogeneous data comparison device in an embodiment of the present invention;

[0022] Figure 6 It is a structural block diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0025] It should be noted that the heterogeneous data comparison method and device of the present invention can be used in data comparison scenarios in the financial field, and can also be used in any field other than the financial field. The embodiments of the present invention do not limit the application field of the heterogeneous data comparison method and device.

[0026] In view of the fact that the existing technology is very likely to cause memory overflow and software crash when comparing large data tables, an embodiment of the present invention provides a heterogeneous data comparison method, which optimizes the strategy for comparing each data in combination with common data scenarios to make the results more accurate and meaningful, and provides comparison of large data tables to solve performance bottlenecks; in the case of batch comparison, it supports the import of parameter files so that the comparison function can be performed automatically and the results can be automatically output, which greatly releases the manpower of manual comparison, and can quickly verify the data table after the migration or synchronization of heterogeneous data sources, providing strong support for the accuracy verification of table data. The present invention is described in detail below in conjunction with the accompanying drawings.

[0027] Existing software cannot support timed or batch comparison and processing, lacks automation support, and is somewhat inadequate for big data comparison scenarios. The present invention can perform timed or batch data comparison on the database after data synchronization migration, such as synchronizing data from the Teradata data warehouse to the MPPDB data warehouse, or synchronizing data from the MPPDB data warehouse to the Teradata data warehouse, etc., and adjusts and optimizes the comparison process to meet the data comparison requirements of the corresponding business scenarios.

[0028] Figure 1 It is a flow chart of a heterogeneous data comparison method in one embodiment of the present invention. Figure 2 It is a flow chart of a heterogeneous data comparison method in another embodiment of the present invention. Figure 3 FIG. 1 is a flow chart of a heterogeneous data comparison method in another embodiment of the present invention. Figures 1 to 3 As shown, the heterogeneous data comparison method includes:

[0029] S101: acquiring two data tables according to a data comparison instruction, and summing and comparing the records in each column of the two data tables according to the data types of the records in each column of the data tables.

[0030] There are differences at the system level in heterogeneous data sources, and existing technologies do not have a good solution for this. For example, data warehouses can store massive amounts of data, which is very different from traditional OLTP (On-Line Transaction Processing). When comparing, it is necessary to conduct a large-scale search and analysis of massive amounts of historical data. Due to the rich and flexible data types, when mapping and comparing the same field, it is necessary to format the field type to ensure the accuracy of the comparison. Existing tools do not have the function of such customized strategies, and are not very flexible.

[0031] To solve the above problems in the prior art, the present invention further includes, before executing S101:

[0032] When starting to use the tool, the user submits configuration parameters to the tool, which include the following information:

[0033] 1. Connection information of heterogeneous data sources: database connection address, user name, password, library name and security authentication, etc.

[0034] 2. List of comparison tables: specify the library name and table name.

[0035] 3. Comparison strategy for corresponding fields in the comparison table: specify comparison of all fields in the table or comparison according to set fields.

[0036] 4. Formatting strategy for comparison data:

[0037] a) Set whether to perform a strict comparison based on the comparison content. If a strict comparison is performed, once any inconsistency occurs, it is considered that there is a difference between the two tables. Otherwise, the subsequent refinement strategy is used for comparison and judgment.

[0038] b) Whether to format certain fields, such as setting the field content of a single database to remove duplicates before comparing it with the corresponding fields in another database table; or setting whether null is treated as equivalent to an empty string; or setting whether to convert the date format, such as considering that dates in the form of yyyy-mm-dd and yyyymmdd are equivalent in content.

[0039] c) Setting for excluding records: some records will have unavoidable differences and need to be ignored or specially processed during comparison so that they are not used as the basis for inconsistency in comparison. Users can ignore this part of the data by setting conditions.

[0040] d) Setting of priority comparison records: In a data table, there are often some key records that determine whether the contents of the table are consistent. In this case, the user is allowed to screen and judge the key records according to specific SQL statements, and then perform subsequent comparisons after meeting specific conditions. For example, the sum of a certain field is set to be compared first, or a certain type of data is screened and then compared first.

[0041] e) Setting of tolerance for record differences: Tolerance represents the proportion of inconsistent records allowed during comparison (i.e. the proportion of inconsistent records to the total number of records). If the tolerance is exceeded, the comparison will be stopped. Otherwise, the comparison will continue until all comparisons are completed, and the inconsistent records will be output to the user. The setting of tolerance gives the user the function of ignoring the inconsistency of the number of records in the data table that may have certain differences, making the comparison more flexible.

[0042] Before summing and comparing the records in each column of the two data tables according to the data types of the records in each column of the data tables, the following steps are also included:

[0043] 1. Database connection: load different types of drivers for different databases to ensure normal connection of the database.

[0044] 2. Compare the number of rows in the table (compare the number of records). The consistency of the number of records is the premise of data consistency, so it is necessary to compare the number of records according to the strategy before starting the actual extraction. If there is a difference in the number of rows, the comparison will be exited and regarded as a difference between the two tables. The result will be output to the user. The efficiency of comparing the number of records is very high, and many unnecessary comparisons can be avoided in advance. If the user removes some records, the comparison of the number of records will give priority to removing the records that do not need to be compared before calculating the statistical quantity, which can more reasonably avoid some inevitable differences in the database.

[0045] 3. Data formatting and ignoring: Extract the fields and records that need to be extracted one by one according to user parameters, and perform corresponding formatting and ignoring processing on the fields that require special processing for subsequent comparison.

[0046] For example, taking the Teradata data warehouse and the MPPDB data warehouse as examples, the following data can be ignored or formatted:

[0047] ① Half a character: When writing data into the Teradata database, if the truncation is not strict (Chinese characters occupy position 2, and odd positions are truncated when truncating pure Chinese characters) or the data exceeds the defined length of the field, the Teradata database will force it to be truncated, resulting in missing encoding of the text content. If the MPPDB database itself does not have this problem during comparison, it will cause differences in the content during comparison. However, from the data level, the two are almost consistent, and the factors that interfere with the comparison need to be eliminated or formatted.

[0048] ② Special characters: Due to differences in encoding, rare characters may appear normally in the original database but appear as garbled characters in the comparison database. Special characters are mostly Chinese characters. For example, the original database uses UTF-8 encoding, while the comparison database is stored in GBK encoding. Since UTF-8 encoding has many more characters than GBK encoding, some characters that only exist in UTF-8 are automatically converted into question marks when entering the GBK-encoded database.

[0049] ③Null and empty string: During the data migration process, the null string may be automatically converted to an empty string or the empty string may be automatically converted to null, which needs to be avoided.

[0050] ④ Rounding: There are differences in the rounding retention mechanisms of the Teradate database and the MPPDB database. In the Teradate database, the rounding limit is .5, while in the MPPDB database, the rounding limit is .4. This scenario needs to be eliminated or formatted.

[0051] 4. Metadata comparison (table structure comparison). Metadata mainly includes the field name, number of fields, and field definition of the table. In the field definition, heterogeneous data sources have underlying differences in data type naming. When the comparison strategy is a strict comparison rule, once there is a difference in metadata information, the comparison will be exited, and the two tables will be considered to be different and the results will be output to the user; otherwise, the user is allowed to set the mapping, that is, a data type in library A is considered to be equivalent to a data type in library B. For example, the float type and double precision type in Teradata are considered to be equivalent to the numeric type and decimal type in MPPDB, respectively.

[0052] According to the data type of each column record in the data table, the records in each column of the two data tables are summed and compared respectively, including:

[0053] The system will prioritize the comparison of some contents according to the priority comparison conditions set by the user. If the conditions are not met, the system will exit the comparison and consider the data inconsistent. The user can specify the SQL statement for extraction, sum the values ​​of a single column (column record), or group a group of data (group by) and then compare the contents in sequence. For example, summing the values ​​of numeric types and grouping and summing the values ​​of time types can greatly reduce the comparison load and determine the differences of certain columns at a very low cost.

[0054] Taking the Teradata data warehouse (database) and the MPPDB data warehouse as examples, the following sum comparison can be performed:

[0055] 1. When the data type of the column record is a numeric value (smallint, integer, decimal, numberic, float, and double precision for the Teradata database, and tinyint, smallint, integer, bigint, and numeric for the MPPDB database), you can first sum and compare all the records in each column (column records) before comparison, and calculate the sum of the values ​​of all the records in each column. If the values ​​are not equal, it can be considered that the two tables have differences in the relevant numeric fields, and the comparison will be exited early.

[0056] 2. When the data type of the column record is character (specifically char, varchar, and longvarchar for Teradata databases, and char, varchar, varchar2, nvarchar2, clob, and text for MPPDB databases), you can take the byte length lengthb of each column record, convert the string content into a numeric length, and then perform a sum comparison. If the overall lengths of the corresponding column records on both sides cannot be consistent, it is considered that the two tables have differences in the relevant character fields, and the comparison is exited early.

[0057] 3. When the data type of the column record is date and time (specifically date for Teradata database, date, time, timestamp and smalldatetime for MPPDB database), you can intercept the column record by year, month or hour (cast operation or substring operation) to obtain the high-dimensional time interval of the specific column, and then group it, filter and count the data in the high dimension, and count the number of column records belonging to a certain year, month and hour for efficient investigation. If the high-dimensional data volume cannot be guaranteed to be consistent, it is considered that there are differences in the relevant date fields of the two tables, and the comparison is exited early.

[0058] S102: Determine a data comparison strategy according to the comparison result of the sum comparison, the unique identifier of the data table, and the data volume.

[0059] In one embodiment, when the comparison results of the records in each column of two data tables are consistent, the unique identifier and data volume of the data table are determined, and the data comparison strategy is determined according to the unique identifier and data volume of each data table.

[0060] Due to the characteristics of massive data in some data warehouses, four comparison strategies can be formulated in the actual data comparison process based on the amount of data and whether sorting by unique identifier of the data table is supported:

[0061] When the data volume is less than the preset threshold and the data table does not have a unique identifier (unique sorting is not supported), the data comparison strategy is the first comparison strategy; when the data volume is less than the preset threshold and the data table has a unique identifier (unique sorting is supported), the data comparison strategy is the second comparison strategy; when the data volume is greater than or equal to the preset threshold and the data table has a unique identifier (unique sorting is supported), the data comparison strategy is the third comparison strategy; when the data volume is greater than or equal to the preset threshold and the data table does not have a unique identifier (unique sorting is not supported), the data comparison strategy is the fourth comparison strategy;

[0062] S103: splicing the records of the data table into corresponding target strings according to the data comparison strategy, and comparing the target strings of the two data tables to generate a data table comparison result.

[0063] In one embodiment, concatenating the records of the data table into corresponding target strings according to the data comparison strategy includes:

[0064] When the data comparison strategy is the first data comparison strategy, all fields of each record are concatenated into a corresponding initial character string; and the characters in the initial character string are sorted according to a preset first sorting rule to obtain a target character string.

[0065] When there is no primary key or unique identifier in the data table, it is difficult to get the corresponding records on both sides according to the sorting order of specific columns. Since the amount of data is not large, all the data can be read into the memory one by one, and all the fields of each record are spliced ​​into a long string, which is compressed by hashing. After reading all the records, an overall sorting is performed with the same rules, and then the sorted results (target strings) of the two databases are read out from the memory one by one. If the target strings of the two data tables are completely consistent or the number of inconsistent target strings does not exceed the preset tolerance (the tolerance is the ratio of the number of inconsistent records to the total number of records), the data of the two data tables can be considered consistent.

[0066] In one embodiment, concatenating the records of the data table into corresponding target strings according to the data comparison strategy includes:

[0067] When the data comparison strategy is the second data comparison strategy, each record of the data table is sorted according to the preset second sorting rule, and all fields of each sorted record are respectively concatenated into a corresponding target character string.

[0068] Since the data table supports unique sorting, the table can be sorted by the database's own sorting logic. On the basis of sorting, the sorted records are read sequentially and stored in memory. In order to avoid encoding problems, they are uniformly converted to bytes format for comparison. The sorted records of the two tables are extracted for full field verification (if there is customized processing of some fields, they are compared after processing) and compared one by one until all data are compared or a certain tolerance is exceeded.

[0069] In one embodiment, concatenating the records of the data table into corresponding target strings according to the data comparison strategy includes:

[0070] When the data comparison strategy is the third data comparison strategy, multiple threads are started to sort the records of the data table according to the preset third sorting rule, and all fields of the sorted records are respectively concatenated into corresponding target strings.

[0071] Since the database supports unique sorting, the table can be sorted using the database's own sorting logic. When the amount of data is too large, it will waste a lot of time to compare each item one by one. For this purpose, multiple threads can be started to extract and sort the results in parallel according to the sorted row records. For example, thread 1 extracts 1-100000 records after sorting, thread 2 extracts 100001-200000 records, and so on. Each thread is responsible for a part of the data in the table, thereby maximizing the efficiency of the comparison.

[0072] The present invention can determine whether to perform pre-sorting before extracting data based on the table space occupied and the system memory size. When the amount of data is too large, directly extracting the data from the memory and then sorting it will cause memory overflow. At this time, the database table can be sorted by file sorting to avoid performance problems of sorting data in memory. Figure 4 FIG. 1 is a flow chart of splicing a target character string in one embodiment of the present invention. Figure 4 As shown, according to the data comparison strategy, each record of the data table is spliced ​​into a corresponding target string including:

[0073] S201: When the data comparison strategy is the fourth data comparison strategy, converting a data table into a data file and dividing the data file into a plurality of divided files.

[0074] Data warehouses have difficulty coping with scenarios where the amount of data is large and there is no unique identifier in the data table. This scenario does not support unique sorting and can only be performed by sorting all fields. The data table will not establish indexes for all fields. If the data is completely extracted into the memory and then sorted, the memory occupied by the program will soar, and memory overflow will easily occur. In response to the above problems, the present invention uses a method of exchanging hard disk space for memory, and exports all records one by one as files, avoiding the memory bottleneck problem; then, the oversized data files are sorted, and a multi-way merge sorting algorithm is used, first using external memory, i.e., the hard disk, to save the intermediate results and then using multi-way merging to sort.

[0075] In specific implementation, the program maintains a fixed buffer in the memory, reads the records in the data file into the buffer one by one, and sorts the extracted contents of the buffer in ascending order when the buffer threshold is exceeded and writes them into the split file, repeating n times until the data in the data file is cut into split files such as f1, f2, f3, ... fn, etc. Subsequent sorting is performed based on the split files.

[0076] S202: Selecting the table minimum record from the file minimum records of each split file, and transferring the table minimum record to the sorting file until the number of records in the sorting file reaches the number of records in the data file.

[0077] In specific implementation, the first line of each split file (the smallest record in the split file) is taken for comparison, and the smallest record (table smallest record) among the n smallest records of the file is transferred to the new file S. The smallest record is taken out from the remaining records of each split file and compared again, until the number of records in the sorted file reaches the number of records in the data file, and the data in the data file is transferred to the sorted file. Through the thinking of "multi-way merging", a completely sorted new file is generated from a very large unordered data file with controllable memory.

[0078] S203: Sort the records in the sort file according to the transfer order, and start multiple threads to splice all the fields of each sorted record into a corresponding target string.

[0079] In specific implementation, multiple threads can be started based on the sorted file, and all fields of each sorted record can be concatenated into corresponding target strings to achieve maximum efficiency.

[0080] Among them, the data table comparison results describe the comparison information in detail, such as the comparison table name, comparison time, data volume, metadata information, field type comparison results and inconsistency ratio. Users can also optimize and adjust according to the table comparison results, and further set a more reasonable comparison strategy to increase the comparison efficiency and benefits. For example, when the comparison time is too long and the difference content is too much to judge, further specify specific comparison content to reduce interference. Or if it is found that some records have only slight differences in format, the formatting conditions of the field can be adjusted later to eliminate the interference part and make the comparison content more high-quality.

[0081] Figure 1 The execution subject of the heterogeneous data comparison method shown can be a computer. Figure 1 As can be seen from the process shown, the heterogeneous data comparison method of the embodiment of the present invention first compares the records of each column of the two data tables according to the data type, and then determines the data comparison strategy according to the comparison result of the sum comparison to splice the records of the data tables into corresponding target strings, compares the target strings of the two data tables to generate a data table comparison result, which can quickly verify the data tables after the migration or synchronization of heterogeneous data sources, solve performance bottlenecks, and improve comparison efficiency.

[0082] The specific process of the embodiment of the present invention is as follows:

[0083] 1. Obtain two data tables according to the data comparison instruction and compare the number of table rows.

[0084] 2. When the number of rows in two data tables is the same, the data in the data tables are formatted and ignored.

[0085] 3. Compare the table structures of the two data tables.

[0086] 4. When the table structures of the two data tables are consistent, sum and compare the records in each column of the two data tables according to the data type of the records in each column of the data tables.

[0087] 5. When the comparison results of the records in each column of the two data tables are consistent, determine the unique identifier and data volume of the data table, and determine the data comparison strategy based on the unique identifier and data volume of each data table.

[0088] 6. According to the data comparison strategy, each record of the data table is spliced ​​into a corresponding target string, and the target strings of the two data tables are compared to generate a data table comparison result.

[0089] In summary, the heterogeneous data comparison method provided by the embodiment of the present invention has the following beneficial effects:

[0090] (1) Perform a complete comparison of data tables after data migration and data synchronization. The comparison strategy is more flexible. The efficiency of the comparison is guaranteed by methods such as early exclusion, key field specification, and elimination of interference records. This realizes user customization requirements that cannot be met by existing tools and can adapt to more business scenarios.

[0091] (2) Quickly identify differences in content storage in data tables based on the data storage characteristics and differences of the database. At the same time, you can also select a more optimized comparison plan based on the strategy and complete a more reasonable verification work in combination with different business scenarios;

[0092] (3) It provides a complete set of comparison strategy analysis, and gives four reasonable solutions for four foreseeable scenarios: sortable small data volume, unsortable small data volume, sortable large data volume, and unsortable large data volume. This not only ensures full utilization of memory but also avoids the technical difficulty of memory overflow through hard disk storage;

[0093] (4) When specifying the comparison strategy, the user is allowed to set the comparison pre-judgment logic based on common sense and business scenarios according to the priority and maximum efficiency principle of the system, so that the comparison steps are more reasonable. In the technical comparison process, priority is given to processing information that can clearly distinguish inconsistencies, such as structure and number of records, to avoid invalid content comparison. For scenarios where data accuracy requirements are not high, tolerance parameters are introduced so that a small amount of data differences during the comparison process will not affect the overall comparison results;

[0094] (5) It solves the pain points of traditional software being inflexible and the inability to manually compare large data tables. It provides sufficient comparison strategies and solutions for databases with large data volumes, which provides good guidance for database migration and data verification after synchronization.

[0095] Based on the same inventive concept, an embodiment of the present invention further provides a heterogeneous data comparison device. Since the principle of solving the problem by the device is similar to that of the heterogeneous data comparison method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0096] Figure 5 is a structural block diagram of a heterogeneous data comparison device in an embodiment of the present invention. Figure 5 As shown, the heterogeneous data comparison device includes:

[0097] A summing and comparing module is used to obtain two data tables according to a data comparison instruction, and to sum and compare the records in each column of the two data tables according to the data types of the records in each column of the data tables;

[0098] A comparison strategy determination module, used to determine a data comparison strategy according to the comparison result of the sum comparison, the unique identifier of the data table and the data volume;

[0099] The comparison result generation module is used to splice each record of the data table into a corresponding target string according to the data comparison strategy, and compare the target strings of the two data tables to generate a data table comparison result.

[0100] In one embodiment, the comparison result generation module includes:

[0101] A string concatenation unit, used for concatenating all fields of each record into corresponding initial strings when the data comparison strategy is the first data comparison strategy;

[0102] The first target character string unit is used to sort the characters in the initial character string according to a preset first sorting rule to obtain a target character string.

[0103] In one embodiment, the comparison result generation module includes:

[0104] a segmentation unit, configured to convert the data table into a data file and segment the data file into a plurality of segmented files when the data comparison strategy is the fourth data comparison strategy;

[0105] A record transfer unit, used for selecting a table minimum record from the file minimum records of each split file, and transferring the table minimum record to the sorting file until the number of records in the sorting file reaches the number of records in the data file;

[0106] The second target string unit is used to sort the records in the sort file according to the transfer order, and start multiple threads to splice all the fields of each sorted record into the corresponding target string.

[0107] In summary, the heterogeneous data comparison device of the embodiment of the present invention first compares the records of each column of the two data tables according to the data type, and then determines the data comparison strategy according to the comparison result of the sum comparison to splice the records of the data tables into corresponding target strings, compares the target strings of the two data tables to generate data table comparison results, and can quickly verify the data tables after the migration or synchronization of heterogeneous data sources, solve performance bottlenecks, and improve comparison efficiency.

[0108] The embodiment of the present invention also provides a specific implementation of a computer device capable of implementing all the steps in the heterogeneous data comparison method in the above embodiment. Figure 6 is a block diagram of a computer device in an embodiment of the present invention, see Figure 6 , the computer equipment specifically includes the following contents:

[0109] Processor (processor) 601 and memory (memory) 602.

[0110] The processor 601 is used to call the computer program in the memory 602. When the processor executes the computer program, all the steps in the heterogeneous data comparison method in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0111] Obtain two data tables according to the data comparison instruction, and sum and compare the records in each column of the two data tables according to the data types of the records in each column of the data tables;

[0112] Determine the data comparison strategy according to the comparison result of the sum comparison, the unique identifier of the data table and the data volume;

[0113] According to the data comparison strategy, each record of the data table is spliced ​​into a corresponding target string, and the target strings of the two data tables are compared to generate a data table comparison result.

[0114] In summary, the computer device of the embodiment of the present invention first sums and compares the records in each column of the two data tables according to the data type, and then determines the data comparison strategy according to the comparison result of the sum and comparison to splice the records in the data tables into corresponding target strings, and compares the target strings of the two data tables to generate data table comparison results. This can quickly verify the data tables after migration or synchronization of heterogeneous data sources, solve performance bottlenecks, and improve comparison efficiency.

[0115] The embodiment of the present invention also provides a computer-readable storage medium capable of implementing all the steps in the heterogeneous data comparison method in the above embodiment. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all the steps in the heterogeneous data comparison method in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0116] Obtain two data tables according to the data comparison instruction, and sum and compare the records in each column of the two data tables according to the data types of the records in each column of the data tables;

[0117] Determine the data comparison strategy according to the comparison result of the sum comparison, the unique identifier of the data table and the data volume;

[0118] According to the data comparison strategy, each record of the data table is spliced ​​into a corresponding target string, and the target strings of the two data tables are compared to generate a data table comparison result.

[0119] In summary, the computer-readable storage medium of the embodiment of the present invention first sums and compares the records in each column of the two data tables according to the data type, and then determines the data comparison strategy according to the comparison result of the sum and comparison to splice the records in the data tables into corresponding target strings, and compares the target strings of the two data tables to generate data table comparison results. This can quickly verify the data tables after migration or synchronization of heterogeneous data sources, solve performance bottlenecks, and improve comparison efficiency.

[0120] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0121] Those skilled in the art may also understand that the various illustrative logical blocks, units, and steps listed in the embodiments of the present invention may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, units, and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present invention.

[0122] The various illustrative logic blocks, or units, or devices described in the embodiments of the present invention can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of the above. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any conventional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other similar configuration.

[0123] The steps of the method or algorithm described in the embodiments of the present invention can be directly embedded in hardware, a software module executed by a processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.

[0124] In one or more exemplary designs, the above functions described in the embodiments of the present invention can be implemented in hardware, software, firmware or any combination of the three. If implemented in software, these functions can be stored on a computer-readable medium, or transmitted in the form of one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media that facilitate the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by any general or special computer. For example, such computer-readable media can include but are not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store program codes in the form of instructions or data structures and other forms that can be read by general or special computers, or general or special processors. In addition, any connection can be appropriately defined as a computer-readable medium, for example, if the software is transmitted from a website site, server or other remote resource through a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wirelessly, such as infrared, wireless and microwave, it is also included in the defined computer-readable medium. The disk and disc include compact disk, laser disk, optical disk, DVD, floppy disk and blue-ray disk. Disks usually copy data magnetically, while discs usually copy data optically with lasers. The above combination can also be included in computer readable media.

Claims

1. A heterogeneous data comparison method, characterized in that: include: Obtain two data tables according to the data comparison instruction, and sum and compare the records in each column of the two data tables according to the data types of the records in each column of the data tables; Determine the data comparison strategy according to the comparison result of the sum comparison, the unique identifier of the data table and the data volume; According to the data comparison strategy, each record of the data table is spliced ​​into a corresponding target string, and the target strings of the two data tables are compared to generate a data table comparison result; The summing and comparing the records in each column of the two data tables according to the data types of the records in each column of the data tables respectively includes: summing and comparing the records in the numerical type and grouping and summing and comparing the records in the time type; The data comparison strategy is determined according to the comparison result of the sum comparison, the unique identifier of the data table and the data volume, including: when the comparison results are consistent, the unique identifier and the data volume of the data table are determined, when the data volume is less than a preset threshold and the data table does not have a unique identifier, the data comparison strategy is a first comparison strategy; when the data volume is less than the preset threshold and the data table has a unique identifier, the data comparison strategy is a second comparison strategy; when the data volume is greater than or equal to the preset threshold and the data table has a unique identifier, the data comparison strategy is a third comparison strategy; when the data volume is greater than or equal to the preset threshold and the data table does not have a unique identifier, the data comparison strategy is a fourth comparison strategy; According to the data comparison strategy, each record of the data table is spliced ​​into a corresponding target string, including: When the data comparison strategy is the first data comparison strategy, all fields of each record are concatenated into corresponding initial character strings; Sorting the characters in the initial character string according to a preset first sorting rule to obtain the target character string; According to the data comparison strategy, each record of the data table is spliced ​​into a corresponding target string, including: When the data comparison strategy is the second data comparison strategy, the records of the data table are sorted according to the preset second sorting rule, and all fields of the sorted records are respectively concatenated into corresponding target strings; According to the data comparison strategy, each record of the data table is spliced ​​into a corresponding target string, including: When the data comparison strategy is the third data comparison strategy, multiple threads are started to sort the records of the data table according to the preset third sorting rule, and all fields of the sorted records are respectively spliced ​​into corresponding target strings; According to the data comparison strategy, each record of the data table is spliced ​​into a corresponding target string, including: When the data comparison strategy is the fourth data comparison strategy, the data table is converted into a data file and the data file is divided into a plurality of divided files; Selecting a table minimum record from the file minimum records of each split file, and transferring the table minimum record to a sorting file until the number of records in the sorting file reaches the number of records in the data file; Sort the records in the sort file according to the transfer order, and start multiple threads to concatenate all the fields of each sorted record into the corresponding target string.

2. A heterogeneous data comparison device, characterized in that: include: A summing and comparing module is used to obtain two data tables according to a data comparison instruction, and sum and compare the records in each column of the two data tables according to the data types of the records in each column of the data tables; A comparison strategy determination module, used to determine a data comparison strategy according to the comparison result of the sum comparison, the unique identifier of the data table and the data volume; A comparison result generating module, used for respectively splicing each record of the data table into a corresponding target string according to the data comparison strategy, and comparing the target strings of the two data tables to generate a data table comparison result; The summing and comparing the records in each column of the two data tables according to the data types of the records in each column of the data tables respectively includes: summing and comparing the records in the numerical type and grouping and summing and comparing the records in the time type; The data comparison strategy is determined according to the comparison result of the sum comparison, the unique identifier of the data table and the data volume, including: when the comparison results are consistent, the unique identifier and the data volume of the data table are determined, when the data volume is less than a preset threshold and the data table does not have a unique identifier, the data comparison strategy is a first comparison strategy; when the data volume is less than the preset threshold and the data table has a unique identifier, the data comparison strategy is a second comparison strategy; when the data volume is greater than or equal to the preset threshold and the data table has a unique identifier, the data comparison strategy is a third comparison strategy; when the data volume is greater than or equal to the preset threshold and the data table does not have a unique identifier, the data comparison strategy is a fourth comparison strategy; The comparison result generation module comprises: A string concatenation unit, used for concatenating all fields of each record into corresponding initial strings when the data comparison strategy is the first data comparison strategy; A first target character string unit, used for sorting the characters in the initial character string according to a preset first sorting rule to obtain the target character string; The comparison result generation module comprises: a segmentation unit, configured to convert the data table into a data file and segment the data file into a plurality of segmented files when the data comparison strategy is the fourth data comparison strategy; A record transfer unit, configured to select a table minimum record from the file minimum records of each segmented file, and transfer the table minimum record to a sorted file until the number of records in the sorted file reaches the number of records in the data file; The second target string unit is used to sort the records in the sort file according to the transfer order, and start multiple threads to splice all the fields of each sorted record into the corresponding target string; According to the data comparison strategy, each record of the data table is spliced ​​into a corresponding target string, including: When the data comparison strategy is the second data comparison strategy, the records of the data table are sorted according to the preset second sorting rule, and all fields of the sorted records are respectively concatenated into corresponding target strings; According to the data comparison strategy, each record of the data table is spliced ​​into a corresponding target string, including: When the data comparison strategy is the third data comparison strategy, multiple threads are started to sort the records of the data table according to the preset third sorting rule, and all fields of the sorted records are respectively spliced ​​into corresponding target strings.

3. A computer device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the heterogeneous data comparison method according to claim 1 are implemented.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the heterogeneous data comparison method according to claim 1 are implemented.

5. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the heterogeneous data comparison method according to claim 1 are implemented.

Images