Data comparison method and device, equipment and storage medium

Abstract

The invention discloses a data comparison method, device and equipment and a storage medium, which can identify different expressions of the same data in different databases by configuring a field type adaptation rule and adapting to special processing of different databases on different data types, and reduce missed report and false report differentiated data caused by data type reasons. By configuring a field mapping rule, the problem of data comparison caused by inconsistency of a source end data table structure and a target end data table structure is solved. The invention provides a feasible, expandable and reliable data comparison method for data comparison after data migration, particularly heterogeneous data comparison after multi-source end data migration.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data comparison method between homogeneous data sources and heterogeneous data sources for target data. Background Art

[0002] During the data migration and synchronization process, ensuring data security and integrity is a major pain point faced by the migration and synchronization work. For the migration and synchronization of millions and tens of millions of MySQL database data, the consistency check of the data can be performed using tools such as mysqlDbCompare to verify the data volume and specific data consistency. However, in many actual data migration processes, the database used for comparison is not necessarily limited to MySQL. mysqlDbCompare can only solve the problem of data comparison between MySQL, but in actual production use, there is a high probability that the source database and the target database are two different databases. Or, due to some specific reasons, the user needs to migrate the original MySQL database data to the Oracle database. Although it is believed that the two tables have a high degree of structural similarity to a certain extent, there are still scenarios where some data comparison tools cannot compare the source database and the target database due to database characteristics. Or, in the data migration process, multiple data migration tools are used to migrate data from different data sources, and the data that needs to be compared includes multiple data source data and one target database data. Faced with the above many complicated scenarios, it is difficult to find a method in the prior art that can solve the above problems at once. Summary of the invention

[0003] The present invention is committed to solving the problem that in the scenario where homogeneous data and heterogeneous data coexist in data comparison, general data comparison methods are unable to complete the complete comparison of homogeneous data and heterogeneous data at the same time, and to solving the problem of missed and false positive differentiated data generated when homogeneous data and heterogeneous data are compared with target end data.

[0004] In order to achieve the above-mentioned object of the invention, the embodiment of the present application first provides a data comparison method, which is characterized by comprising: Configure field type adaptation rules; Configure field mapping rules; Obtain the data to be compared from the source database and the target database; By matching the source database with the target database data table, the data comparison task is split into the data table comparison task; Obtain metadata information of the data table to be compared; Filter the field mapping rules corresponding to the data table to be compared, and generate a field mapping list corresponding to the data table to be compared; Find the corresponding comparison column according to the field mapping list; Non-heterogeneous data is directly compared, and heterogeneous data is compared field by field based on field type adaptation rules and field mapping rules.

[0005] In order to achieve the above-mentioned object of the invention, the embodiment of the present application further provides a data comparison device, which is characterized by comprising: Metadata manager, used to uniformly manage metadata information of the data to be compared; Data field adapter, used to adapt field types according to field type adaptation rules during data comparison; Mapping rule adapter, used to adapt fields according to field mapping rules during data comparison; The data comparer is used to compare data based on the field type adaptation results and field adaptation results.

[0006] To achieve the above-mentioned purpose of the invention, an embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the memory stores a program that can be run on the processor, and is characterized in that when the program is executed by the processor, the steps of the above-mentioned data comparison method are implemented.

[0007] Also, a computer-readable storage medium is provided, storing at least one program, characterized in that the at least one program can be executed by at least one processor to implement the steps of the above-mentioned data comparison method.

[0008] The data comparison method of the present application is applicable to the data comparison problem between homogeneous data sources and heterogeneous data sources for target data, and in particular, provides a comprehensive adaptation solution for heterogeneous data comparison in unconventional migration scenarios of multi-data source migration. The data comparison method of the present application configures field type adaptation rules to adapt to the special processing of different databases for different data types, and can identify the different manifestations of the same data in different databases, reducing the omissions and false positives of differentiated data due to data types; by configuring field mapping rules, the data comparison problem caused by the inconsistency between the source data table structure and the target data table structure is solved. The data comparison method of the present application can solve the problems encountered in most heterogeneous data comparison scenarios, realize data comparison between different databases, data comparison between data tables with different data structures, and data comparison within a specific range. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a flow chart of the data comparison method of Embodiment 1 of the present application; Figure 2 This is a logical diagram of a data comparison process flow of a data comparison device according to Embodiment 2 of the present application; Figure 3This is a schematic diagram of the full data comparison process of heterogeneous data in Example 3 of the present application; Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0010] In order to make the technical solution of the present invention more clearly understood, the technical solution of the present invention is further described in detail below according to specific embodiments and in combination with the accompanying drawings. Obviously, the following embodiments are only part of the embodiments of the present invention and do not constitute a limitation on the technical solution of the present invention. For those of ordinary skill in the art, other embodiments can also be obtained according to the embodiments of the present application without paying creative labor.

[0011] It should be noted that the order of steps described in the method embodiments in the following embodiments does not constitute a limitation on the technical solution of the present invention. For ordinary technicians in this field, without paying any creative labor, they can also reasonably adjust the order of certain steps according to the embodiments of the present application. Example 1

[0012] This embodiment provides a data comparison method, which is based on the fact that the source database and the target database have completed data migration through basic migration tools. The method is suitable for data comparison scenarios between homogeneous or heterogeneous databases and multiple source terminals, and can realize full consistency verification of the source database and the target database.

[0013] The data comparison method of this embodiment is as follows: Figure 1 As shown, the specific steps include: Step S101: configure field type adaptation rules.

[0014] Configuring field type adaptation rules includes the following two aspects: (1) Configure adaptation rules for different databases to handle the same type of data in different ways.

[0015] In data comparison, different databases may process the same data type differently. For example, in MySQL database, the blob data type is used to store binary data, while in postgres, the bytea data type is used to store binary data. For these two data types, it is necessary to identify and mark the field information in the metadata, configure the adaptation rules of the two databases for the different storage methods of binary data, that is, the adaptation relationship between the blob data type and the bytea data type in the two databases, and realize the correct comparison of the two types of data, the blob data type and the bytea data type, in the subsequent comparison, to solve the problem of false positives and differentiated data caused by data comparison between different databases.

[0016] (2) Configure adaptation rules for different databases to process different types of data.

[0017] In data comparison, in addition to the scenario where different databases handle the same data type differently, there will also be extreme cases where the source field is char and the target field is varchar. However, for some databases, the char type can be configured to change its data acquisition logic. For example, 20-bit char type data stores 10 digits. When acquiring data, there may be a possibility that 10 digits have 10 spaces, which has a greater impact on the data comparison results of the two databases. Alternatively, for the comparison of time types, it is necessary to consider the impact of time precision on the comparison results. By configuring different databases to pay attention to time precision for this type of data, the problem of data misreporting caused by different precision can be solved.

[0018] The above descriptions are several examples of configuring field type adaptation rules, and are not intended to limit the configuration of field type adaptation rules. In response to the growing demand for data comparison, by adding special adaptations for different data types, the needs of data migration comparison facing heterogeneous data structures can be met. When comparing the same data type, the data retrieval logic and comparison are performed normally. When comparing different data types, the retrieved data can be processed again according to the field type adaptation rules to make the data on both the source and target ends consistent before comparison.

[0019] Step S102: configure field mapping rules.

[0020] Configuring field mapping rules is used to solve the data comparison problem when the source data structure and the target data structure are partially or completely heterogeneous. This scenario is different from the heterogeneous data structure scenario for configuring field type adaptation rules. The heterogeneous data structure difference problem solved by configuring field type adaptation rules only includes the difference in field types in the data table, while configuring field mapping rules solves the problem of differences in table structures between the source and target data tables and comparison of data with different field names on the source and target sides.

[0021] There are two situations. The first one is that the fields with different field names on the source and target have the same data meaning and content, so the two fields can be directly matched for comparison. The second one is that the fields with different field names on the source and target have inconsistent data meanings, so the content needs to be converted to a certain extent before comparison.

[0022] Therefore, in this embodiment, configuring the field mapping rule includes: (1) Configure field mapping relationships for fields with different source and target field names but the same data meaning and content; (2) Configure field mapping relationships and content conversion rules for fields that need to be converted because the source and target field names are different and the data meanings are inconsistent.

[0023] Field mapping rules are stored in the form of key-value pairs, which include (1) the mapping relationship between source fields and target fields; and (2) the conversion rules between source fields and target fields.

[0024] The above steps S101 and S102 are preparatory steps for data comparison, which can be completed by the user through the operation interface, or can be configured in advance in the system.

[0025] Step S103: Obtain data to be compared from the source database and the target database.

[0026] Step S104: split the data comparison task into data table comparison tasks by matching the source database with the target database data table.

[0027] Specifically, the data comparison task is split into data table comparison tasks, which are sent to the thread pool for processing. Each thread can execute the data table comparison tasks in parallel.

[0028] Step S105: Obtain metadata information of the data table to be compared.

[0029] Specifically, each thread that executes the data table comparison task obtains the metadata information of the table that needs to be compared from the metadata manager, and obtains the field information by calling the getMetaData method: field name, field length, field type, field precision, primary key information, index information. The obtained information is stored in a unified global cache for subsequent comparison or other uses. After obtaining the field data in the database, the select statement of the query statement can be spliced. At the same time, through the identification mark of special field types in the metadata manager, the columns that need special queries can be obtained. For example, for blob columns, the value needs to be obtained after hex to solve the problem of binary data in the comparison process.

[0030] Step S106: Filter the field mapping rules corresponding to the data table to be compared, and generate a field mapping list corresponding to the data table to be compared.

[0031] Before data comparison, first filter out the field mapping rules corresponding to the data table to be compared from the configured field mapping rules, generate the field mapping lists of the source and target ends, record them with the data table names, build the comparison mapping relationship between the source and target fields, and add content conversion rules for the fields that need to be converted. The content conversion rules include truncation, splicing, four arithmetic operations, etc., which can be configured by the user.

[0032] Step S107: searching for corresponding comparison columns according to the field mapping list.

[0033] During the data comparison process, first find the corresponding matching rule based on the source table name. One source field may correspond to one target field, or may correspond to multiple target fields. Therefore, in the field mapping list, the stored target field is a target column set. After finding the corresponding target column set, the corresponding content conversion rule can be found from the corresponding mapping relationship. If there is no content conversion rule, you only need to directly compare the source and target values. If there is a content conversion rule, you need to convert the source data according to the content conversion rule and compare it with the target.

[0034] Step S108: For non-heterogeneous data, a comparison is directly performed field by field; for heterogeneous data, a comparison is performed field by field according to field type adaptation rules and field mapping rules.

[0035] In the case of a large amount of data, in order to improve the efficiency of data comparison, the data can be split to reduce the amount of data compared in each batch. As a preferred solution, the data comparison method of the above embodiment can further include the following steps after step S105 and before step S106: according to the distribution of the data to be compared, the data to be compared is evenly split into several data segments, and each data segment is compared synchronously.

[0036] The differences in structure of heterogeneous data sources can be solved by configuring field mapping rules. The differences in data can be compared after data screening. For example, in the actual migration process, multiple source data sources output data to the same database, but in the actual storage structure of the target database, the source data is distinguished by customer number. If you simply compare the data of source end one with the data of the target end, the data of source end two that falls into the target end will be considered as differentiated data. When the data of source end two is compared with the data of the target end, the data of source end one that falls into the target end will be considered as differentiated data, resulting in false positive differences.

[0037] Therefore, in the scenario where multiple source databases migrate data to the same target database, the data comparison method of the above embodiment may further include the following steps: by configuring data screening rules, comparing data within a limited range of the source database and / or the target database.

[0038] Data filtering is applied in the query SQL assembly stage. During the data comparison process, data filtering is added to each batch of SQL generated. The filtering rules are divided into rules for the source end, rules for the target end, and rules for both sides to cope with asymmetric data comparison scenarios. First, under the normal SQL assembly logic, determine whether there are data filtering rules. If there are filtering rules, record the filtering rules under the corresponding table model. When assembling each batch of SQL statements on the source and target ends, determine whether the applied rules are applied to the source end or the target end, or to both sides. For the identified filtering rules, after splicing them into the query SQL statements on the source or target end, in the query stage, the data is restricted to a certain range for query. When comparing data, data comparison will only be performed within a limited range.

[0039] The innovative points of this embodiment method are as follows: Adaptation of heterogeneous data source types: Different from the existing data comparison method, the data comparison method of this embodiment adapts to the special processing of different data types in different databases by configuring field type adaptation rules. It can identify the different manifestations of the same data in different databases, and can perform data comparison on heterogeneous data sources. It can completely and correctly compare data from multiple databases such as MySQL, Oracle, DB2, PostgreSQL, etc., reducing the missed and false positives of differentiated data due to data types.

[0040] Adaptation of heterogeneous data structures: In many actual data migration and synchronization processes, there are often cases where the source data table structure is inconsistent with the target data table structure. For general data comparison devices, even if the data sources are homogeneous, this type of problem still exists. Existing data comparison methods cannot solve the problem of different data structures. The data comparison method of this embodiment solves the data comparison problem caused by the inconsistency between the source data table structure and the target data table structure by configuring field mapping rules.

[0041] Comparison and adaptation of derived data: For some special scenarios, if data rows need to be specially marked, fields are often added to classify the data. The value of this field may come from other fields of the same or different types. Usually, it is converted and migrated through functions, and the attribute field does not actually exist in the source database. The data comparison method of this embodiment can add mapping rules to the source attributes by configuring field mapping rules to achieve comparison of derived data.

[0042] Screening and comparison mechanism for multi-source data: The data comparison method of this embodiment is a method for comparing source data and target data in a multi-source data migration scenario. A data screening mechanism is introduced in the data comparison process to solve the problem of introducing the same attribute data from different sources during the migration process.

[0043] The data comparison method of this embodiment is applicable to the data comparison problem between homogeneous data sources and heterogeneous data sources for target data, especially for heterogeneous data comparison in unconventional migration scenarios of multi-data source migration, and provides a comprehensive adaptation solution that can solve the problems encountered in most heterogeneous data comparison scenarios, and realize data comparison between different databases, data comparison between data tables with different data structures, and data comparison within a specific range. Due to the ever-changing types and data types of databases, emerging databases have emerged in recent years. For source or target data types not mentioned in this embodiment, they can be expanded and applied according to the method mentioned in this embodiment. Example 2

[0044] This embodiment provides a data comparison device, such as Figure 2 As shown, including: The metadata manager is used to manage the metadata information of the data to be compared.

[0045] An important step in processing heterogeneous data is to manage heterogeneous data sources. In the complex database market, the most difficult thing is to manage the database in a unified way. There is the sqlyog management tool for MySQL, the sqlplus tool for Oracle, and a unified database management tool such as navicat, which can handle metadata information of different databases well. The metadata manager of this embodiment manages metadata in an effective and unified way, and provides data field name information, field type information, field precision information, etc. for subsequent data comparison.

[0046] The metadata manager adopts an interface-based design. Different databases are connected to the metadata manager through extended interfaces to implement metadata database acquisition methods, data retrieval methods, etc. Various database implementation classes implement the getMetaData method of the database management class, which contains database field information: field name, field precision, field type, field length, etc., as well as the primary key information of the table, the index information acquisition method of the table, and the method of obtaining detailed database data provided to the outside world.

[0047] Data field adapter, used to adapt field types according to field type adaptation rules during data comparison. Field type adaptation rules include the following two aspects: (1) Adaptation rules for different databases to handle the same type of data in different ways.

[0048] In data comparison, different databases may process data of the same data type differently. For example, in MySQL database, binary data is stored using the blob type, and in postgres, binary data is stored using the bytea type. To compare these two types of data, it is necessary to identify and mark the field information in the metadata manager. The data field adapter can configure the adaptation rules of the two databases for different storage methods of binary data, that is, the adaptation relationship between the blob data type and the bytea data type in the two databases. In subsequent comparisons, the correct comparison of the two types of data, the blob data type and the bytea data type, can be achieved, solving the problem of false positives and differentiated data caused by data comparisons between different databases.

[0049] (2) Adaptation rules of different databases for processing different types of data.

[0050] In the data comparison process, in addition to the scenario where different databases handle the same data type differently, there will also be extreme cases where the source field is char and the target field is varchar. However, for some databases, the char type can be configured to change its data acquisition logic. For example, 20-bit char type data stores 10 digits. When acquiring data, there may be a possibility that 10 digits have 10 spaces, which has a greater impact on the data comparison results of the two databases. Alternatively, for the comparison of time types, it is necessary to consider the impact of time precision on the comparison results. The data field adapter can solve the problem of data misreporting differences caused by different precision by configuring whether to pay attention to time precision.

[0051] The above describes several examples of field type adaptation by data field adapters, which are not intended to be a restriction on field type adaptation by data field adapters. The data field adapter adopts an interface-based design. In response to the growing demand for data comparison, special adaptations for different data types are added to meet the data comparison needs when facing heterogeneous data structures. For different data types, a strategy pattern is used to manage them: when comparing the same data type, the data retrieval logic and comparison are performed normally. When comparing different data types, the retrieved data is processed again according to the field adaptation results of the data field adapter, so that the data on both ends of the source and target are presented in the same form before comparison.

[0052] Mapping rule adapter, used to perform field adaptation according to field mapping rules during data comparison.

[0053] The mapping rule adapter is used to solve the situation where the source data structure and the target data structure are partially or completely heterogeneous. This scenario is different from the heterogeneous data structure scenario oriented by the data field adapter. The heterogeneous data structure difference problem solved by the data field adapter through field type adaptation only includes the difference in field type in the data table, while the mapping rule adapter solves the problem of difference in table structure between the source and target data tables and comparison of data with different field names on the source and target sides through field adaptation.

[0054] There are two situations. The first one is that the fields with different field names on the source and target have the same data meaning and content, so the two fields can be directly matched for comparison. The second one is that the fields with different field names on the source and target have inconsistent data meanings, so the content needs to be converted to a certain extent before comparison.

[0055] The mapping rule adapter stores field mapping rules in the form of key-value pairs, which include (1) the mapping relationship between source fields and target fields; and (2) the conversion rules between source fields and target fields.

[0056] Before data comparison, the mapping rule adapter first filters out the field mapping rules corresponding to the data table to be compared from the field mapping rules, generates the field mapping lists of the source and target ends, records them with the data table names, builds the comparison mapping relationship between the source and target fields, and adds content conversion rules to the fields that need to be converted. The content conversion rules include truncation, splicing, four arithmetic operations, etc., which can be configured by the user.

[0057] During the data comparison process, the mapping rule adapter first finds the corresponding matching rule based on the source table name. One source field may correspond to one target field, or may correspond to multiple target fields. Therefore, in the field mapping list, the stored target field is a target column set. After finding the corresponding target column set, the corresponding content conversion rule can be found from the corresponding mapping relationship. If there is no content conversion rule, you only need to directly compare the source and target values. If there is a content conversion rule, you need to convert the source data through the content conversion rule and compare it with the target.

[0058] The data comparer is used to compare data based on the field type adaptation results and field adaptation results.

[0059] The differences in the structure of heterogeneous data sources can be solved by using mapping rule adapters, and the differences in data can be compared after data screening. For example, in the actual migration process, multiple source data sources output data to the same database, but in the actual storage structure of the target database, the source data is distinguished by customer numbers. If you simply compare the data of source end one with the data of the target end, the data of source end two that falls into the target end will be considered differentiated data. When the data of source end two is compared with the data of the target end, the data of source end one that falls into the target end will be considered differentiated data, resulting in false positive differences.

[0060] Therefore, in the scenario where multiple source databases migrate data to the same target database, the data comparison device of the above embodiment may further include a data filter for filtering data within a limited range from the source database and / or the target database for comparison according to data filtering rules.

[0061] The data filter is applied to the query SQL assembly stage. During the data comparison process, the data filtering application is added to each batch of SQL generated. The filtering rules are divided into rules for the source end, rules for the target end, and rules for both sides to cope with asymmetric data comparison scenarios. First, under the normal SQL assembly logic, determine whether there are data filtering rules. If there are filtering rules, record the filtering rules under the corresponding table model. When assembling each batch of SQL statements on the source and target ends, determine whether the applied rules are applied to the source end or the target end, or to both sides. For the identified filtering rules, after splicing them into the query SQL statements on the source or target end, the data will be restricted to a certain range in the query stage. When comparing data, data comparison will only be performed within a limited range.

[0062] For the data migration scenario of multiple sources, after the migration tool uniformly migrates multiple types of data sources such as data source 1, data source 2, etc. to the target data source, data comparison faces the problem of different data types at the source end. At the same time, for different data source types, what needs to be solved is the difference in data types. To this end, the data comparison device of this embodiment manages the metadata of the database through an interface through a metadata manager, abstracting a unified interface. For the data source connected to the data comparison device, it is only necessary to implement a unified interface in the metadata manager to complete the docking of the source end, effectively providing the expandability of the data comparison device. Through the data field adapter, the comparison processing of different database field data is completed, and the data field adapter also adopts the same interface form, which is convenient for expansion and updating. In the scenario where data from multiple source data sources are migrated to the same target database, the amount of data at the target end will far exceed the amount of data at the source end. According to the general data comparison device, a large amount of database resources are required to be used for comparison, and redundant data is queried and more CPU is required for data comparison operations. By setting a data filter, the data can be concentrated in the data blocks that need to be compared, reducing the database pressure and CPU pressure.

[0063] The data comparison device of this embodiment performs field adaptation through a mapping rule adapter, so that the data comparison device can support full static data comparison for data migration between heterogeneous data sources without false positives or false negatives. For data comparison of homogeneous data sources, no configuration is required, and the data comparison device can automatically identify the columns that need to be compared. Whether it is a migration scenario with multiple source terminals of the same tool, or a scenario where multiple migration tools migrate the same target library data, the data comparison task can be completed accurately through the coordination of the metadata manager, data field adapter, mapping rule adapter, and data filter. Example 3

[0064] like Figure 3 The figure shows a flow chart of applying the data comparison method of the present application to the full data comparison of heterogeneous data. The flow is based on the data migration work that has been completed on the source and target ends through basic migration tools, and the following is the data comparison work.

[0065] First, the source database and the target database are split into table comparison tasks through data table matching. Several comparison tasks are sent to the thread pool for processing. Each thread obtains the metadata information of the table to be compared from the metadata manager, and obtains the field information by calling the getMetaData method: field name, field length, field type, field precision, primary key information, index information. This type of information is stored in a unified global cache for subsequent comparison or other uses. After obtaining the field data in the database, the select statement of the query statement can be spliced. At the same time, the columns that need special queries can be obtained here. For example, for blob columns, the value needs to be obtained after hex to solve the problem of binary data in the comparison process.

[0066] Then, in the comparison task of the table, first parse the configured mapping rule information through the mapping rule adapter, and filter out the mapping rules corresponding to the comparison task of the table. In this step, the rule information is saved in a map cache, and the corresponding rules can be directly obtained through the table name. At the same time, if the table is configured with rules such as interception and four arithmetic operations, the interception method corresponding to the field can be directly obtained. The interception method may include the parameters of where to intercept and how many characters, and the four arithmetic operations may include the data of the field plus a certain number of digits. The method needs to be retained under the corresponding table field mapping rule for use when executing specific data comparison methods.

[0067] In the data splitting stage, the data will be evenly split into several segments according to the data distribution of the whole table. For example, if the primary key id in the database table ranges from 1 to 10w, the data of 1-10w will be split into 1026 even segments by the primary key in the splitting stage, and these 1026 even segments will be processed by different threads to improve the efficiency of data query. After the data splitting process, the configured data filtering rules will be parsed in the process of assembling the data query statement. This step is completed by the data filter. When the data filter parses the filtering rules, it is applied on the source side or the target side or bilaterally. The filtering rules are bound to the completed SQL assembly task and added to the query statement of each segment. For example, if the filtering rule is id <5000, the data behavior to be compared is the data with id <5000. If the source side is applied and the target side is not applied, the data with id from 1-5000 on the source side will be compared with the data with id from 1-10w on the target side. On the contrary, if the source side does not apply it but the target side applies it, the data of 1-10w on the source side will be compared with the data of 1-5000 on the target side. After completing the SQL assembly task, the assembled SQL will be sent to the query task for data query operation. The data query operation is also performed according to the divided segments.

[0068] In the data comparison phase, first of all, the configured field mapping rules need to be parsed. This step is completed by the mapping rule adapter. For each column on the source side, the corresponding comparison column is found through the cache of the field mapping rules. If the corresponding column is not found, it means that the column names compared on the target side and the source side are consistent. If the field mapping rules are configured, multiple corresponding comparison column sets B may be found. In set B, there may be content conversion rule operations for this column.

[0069] When performing data comparison, the data comparer takes a piece of data from the source table, starts to compare fields one by one, obtains the corresponding field of the target from set B for comparison, obtains the data type of the field, and then obtains the comparison method from many comparison scenarios through the data field adapter, such as the comparison method of string type, the comparison method of time type, the comparison method of numeric type, etc. Then, the data comparer executes the specific comparison method based on the comparison method read by the data field adapter and the information such as the number of digits of precision that may be configured. If there is a field difference between the source data and the target data, the data comparer will record the difference data. Since the field mapping rules or field conversion rules may be configured, the differentiated source data and the corresponding conversion rules will be recorded when printing the log of differentiated data. Example 4

[0070] This embodiment provides an electronic device, such as Figure 4 As shown, it includes a processor 31 and a memory 32 that is in communication with the processor 31. The memory 32 stores a program that can be run on the processor 31, and when the program is executed by the processor 31, the steps of the data comparison method provided in the above embodiment are implemented.

[0071] The above program can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. Example 5

[0072] This embodiment provides a computer-readable storage medium, which stores at least one program. The at least one program can be executed by at least one processor to implement the steps of the data comparison method provided in the above embodiment.

[0073] The above storage media include: U disk, mobile hard disk, read-only memory, random access memory, disk or optical disk and other media that can store program code and their combination. With the development of science and technology, the meaning of storage media may become more and more extensive and not limited to tangible media.

[0074] Those skilled in the art can understand that the above-mentioned embodiments are specific embodiments for implementing the present invention. In addition to the above-mentioned embodiments, the present invention may also have other implementation modes; any technical solutions formed by equivalent replacement or equivalent transformation shall fall within the protection scope required by the present invention.

Claims

1. A data comparison method, characterized in that: include: Configure field type adaptation rules; Configure field mapping rules; Obtain the data to be compared from the source database and the target database; By matching the source database with the target database data table, the data comparison task is split into the data table comparison task; Obtain metadata information of the data table to be compared; Filter the field mapping rules corresponding to the data table to be compared, and generate a field mapping list corresponding to the data table to be compared; Find the corresponding comparison column according to the field mapping list; Non-heterogeneous data is directly compared, and heterogeneous data is compared field by field based on field type adaptation rules and field mapping rules.

2. The data comparison method according to claim 1, characterized in that: The configuration field type adaptation rules include: Configure the adaptation rules for different databases to handle the same type of data in different ways; Configure the adaptation rules for different databases to process different types of data.

3. The data comparison method according to claim 1, characterized in that: The configuration field mapping rules include: Configure field mapping relationships for fields with different source and target field names but consistent data meanings and contents; Configure field mapping relationships and content conversion rules for fields that need to be converted because the source and target field names are different and the data meanings are inconsistent.

4. The data comparison method according to claim 1, characterized in that: After obtaining the metadata information of the data table to be compared, the method further includes: evenly splitting the data to be compared into a plurality of data segments according to the distribution of the data to be compared, and comparing each data segment synchronously.

5. The data comparison method according to claim 1, characterized in that: Also includes: Configure data filtering rules to compare data within a specified range of the source database and / or target database.

6. The data comparison device according to claim 1, characterized in that: include: Metadata manager, used to uniformly manage metadata information of the data to be compared; Data field adapter, used to adapt field types according to field type adaptation rules during data comparison; Mapping rule adapter, used to adapt fields according to field mapping rules during data comparison; The data comparer is used to compare data based on the field type adaptation results and field adaptation results.

7. The data comparison device according to claim 6, characterized in that: The field type adaptation rules include: Adaptation rules for different databases’ processing methods for the same type of data; Adaptation rules for different databases to process different types of data.

8. The data comparison device according to claim 6, characterized in that: The field mapping rules include: The mapping relationship between the fields with different source and target field names but the same data meaning and content; The mapping relationship and content conversion rules of fields that need content conversion when the source and target field names are different and the data meanings are inconsistent.

9. The data comparison device according to claim 6, characterized in that: Also includes: The data filter is used to filter data within a limited range from the source database and / or the target database for comparison according to data filtering rules.

10. An electronic device comprising a processor and a memory, wherein the memory stores a program that can be run on the processor, characterized in that: When the program is executed by the processor, the steps of the data comparison method according to any one of claims 1 to 5 are implemented.

11. A computer-readable storage medium storing at least one program, characterized in that: The at least one program can be executed by at least one processor to implement the steps of the data comparison method described in any one of claims 1 to 5.

Images