Metadata-based data comparison method

Abstract

The invention discloses a data comparison method based on metadata. The method specifically comprises the following steps: acquiring data structures of a source database and a target database to formulate corresponding meta-models; reading metadata of the source database and the target database according to the meta-model and the connection information of the database, and recording the metadata into an intermediate database for comparison; on the basis of the total table record number, the field data type and the like in the metadata, determining the size of a fragment modulo cardinal number, calculating a hash value of a unique identification key by adopting an XXHash algorithm, and determining the number of fragments, the serial number of each fragment and the hash value range corresponding to each fragment; the XXHash algorithm is adopted again to calculate the data of the corresponding fragments, and corresponding hash values are generated; and comparing detailed data in the corresponding fragments, and outputting a difference result. The method has the advantages that the data processing process is optimized, the high calculation speed is kept, the overall efficiency is effectively improved, and two or more data sets can be efficiently and accurately compared.

Description

Technical Field

[0001] The present invention relates to the technical field related to data processing, and in particular to a data comparison method based on metadata. Background Art

[0002] With the development of information technology, data volumes are growing exponentially. Extracting valuable information from this massive amount of data quickly and efficiently has become a crucial issue. Data comparison is a common and critical component of many data processing tasks. Existing data comparison methods often suffer from inefficiencies and inaccuracies, particularly when dealing with large datasets. Summary of the Invention

[0003] The present invention aims to overcome the above-mentioned deficiencies in the prior art and provides a metadata-based data comparison method that can perform efficient and accurate comparison.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions: A data comparison method based on metadata specifically comprises the following steps: (1) Obtain the data structure of the source database and the target database, and formulate a corresponding meta-model based on the data structure, where the meta-model is used to describe the metadata attributes of the database and their relationships; (2) Read the metadata of the source database and the target database according to the defined meta-model and database connection information, record the metadata in the intermediate database, compare the metadata in the intermediate database, and output the difference results; (3) Based on the total number of table records, field data types, and other relevant factors in the metadata, determine the shard modulus cardinality size and use the XXHash algorithm to calculate the hash value of the unique identification key; (4) Determine the number of shards, the number of each shard, and the hash value range corresponding to each shard based on the total number of table records, the shard modulus base, and the hash value in the metadata; (5) Read the data in the source database and the target database, use the XXHash algorithm to calculate the data of the corresponding shards, generate the corresponding hash value, and record the hash value in the record table of the intermediate database; (6) Based on the shard number, read the data in the source database and the data in the target database, compare the detailed data in the corresponding shards, compare each field of each record one by one, and output the difference results.

[0005] In this method, the total number of table records and the characteristics of field data types significantly impact overall processing efficiency. Therefore, it is necessary to determine the size of the shard modulus base based on these factors to optimize data processing. Furthermore, the use of the non-secure XXHash algorithm not only maintains high computational speed when processing large amounts of data, but also effectively improves overall efficiency by reducing unnecessary computational complexity, enabling efficient and accurate comparison of two or more datasets.

[0006] Preferably, in step (1), specifically: a meta-model is formulated according to the data structure of the source database, and the meta-model of the source database describes the metadata attributes and the relationships of the source database; another meta-model is formulated according to the data structure of the target database, and the meta-model of the target database describes the metadata attributes and the relationships of the target database.

[0007] Preferably, when the source database type or the target database type is a relational database, a metamodel applicable to the data structure and relational model of the relational database is formulated; when the source database type or the target database type is a non-relational database, a metamodel applicable to the data structure and storage model of the non-relational database is formulated.

[0008] Preferably, in step (2), the connection information refers to the information required by the program to access the corresponding database; the intermediate database is an independent database, and the program's persistent data needs to be stored in the intermediate database; specifically: the metadata of the source database and the metadata of the target database are recorded in the intermediate database, and the metadata of the source database recorded in the intermediate database are compared with the metadata of the target database. If there is a difference, the difference result is output. Since the data structure of the source database is inconsistent with the data structure of the target database, the data content cannot be compared, and the entire data comparison process ends; if there is no difference, proceed to the next step.

[0009] Preferably, the metadata refers to data used to describe data, including database information, data table information, data field information, the total number of table records and the table unique identification key; when the metadata of the source database completely matches the metadata of the target database, it indicates that the two have the same structural characteristics, and the subsequent data comparison process continues to be executed; conversely, if there is a difference between the metadata of the source database and the metadata of the target database, it indicates that the two represent different entities or structures, and no further comparison operation is required.

[0010] Preferably, the intermediate database is a relational database, which is used to store detailed information related to each shard, including the shard number, the name of the database to which the shard belongs, the name of the data table to which the shard belongs, the unique identification key of the shard, the value range of the unique identification key, and the hash value calculated by the unique identification key using the XXHash algorithm.

[0011] Preferably, in step (3), other relevant factors refer to: memory occupied by program processing, number of shards, total space occupied by each piece of data, total number of records in a table, and space occupied by a table; the calculation method of the shard modulo cardinality adopts one of the following methods: shard modulo cardinality = (memory occupied by program processing / number of shards) / total space occupied by each piece of data; or, shard modulo cardinality = (memory occupied by program processing / number of shards) / (space occupied by a table / total number of records in a table); wherein the shard modulo cardinality is rounded down, the memory occupied by program processing refers to the memory occupied by the program during operation, the total number of records in a table refers to the total number of records in the table, and the space occupied by a table refers to the disk space occupied by a table stored in the database.

[0012] Preferably, the specific operation method for summing up the space occupied by each piece of data is: query the table structure to obtain the data type and length of each field in the table, calculate the size of each field according to the data type and length, and add up the size of each field to obtain the total size of each piece of data.

[0013] Preferably, in step (6), specifically: if the hash values of the shards are consistent, it is determined that the data contents of the corresponding shards in the source database and the target database are consistent; if the hash values of the shards are inconsistent, the corresponding detailed data are read from the source database and the target database respectively according to the shard number, database name, data table name, unique identification key and its value range; then, each field of each record in the shard is compared one by one to check whether there are any differences, and a detailed difference result is output.

[0014] The beneficial effects of the present invention are: determining the size of the shard modulus base based on the total number of records in the table and the field data type to optimize the data processing process; adopting the non-secure encryption XXHash algorithm, not only can a high computing speed be maintained when processing large amounts of data, but also the overall efficiency can be effectively improved by reducing unnecessary computing complexity, thereby enabling efficient and accurate comparison of two or more data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a schematic diagram of the comparison data structure; Figure 3This is a schematic diagram of the structure after sharding; Figure 4 This is a structural diagram after data comparison is inconsistent. DETAILED DESCRIPTION

[0016] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0017] like Figure 1 In the embodiment described, a metadata-based data comparison method specifically includes the following steps: (1) Obtain the data structure of the source database and the target database, and formulate a corresponding meta-model based on the data structure, where the meta-model is used to describe the metadata attributes of the database and their relationships; Specifically: a metamodel is formulated based on the data structure of the source database, and the metamodel of the source database describes the metadata attributes and their relationships of the source database; another metamodel is formulated based on the data structure of the target database, and the metamodel of the target database describes the metadata attributes and their relationships of the target database.

[0018] When the source database type or the target database type is a relational database, a metamodel applicable to the data structure and relational model of the relational database is formulated; when the source database type or the target database type is a non-relational database, a metamodel applicable to the data structure and storage model of the non-relational database is formulated.

[0019] (2) Read the metadata of the source database and the target database according to the defined meta-model and database connection information, record the metadata in the intermediate database, compare the metadata in the intermediate database, and output the difference results; Connection information refers to the information required by the program to access the corresponding database, such as the database's IP address, port number, user name, password, etc.; the intermediate database is an independent database, and the program's persistent data needs to be stored in the intermediate database.

[0020] Specifically: the metadata of the source database and the metadata of the target database are recorded in the intermediate database, and the metadata of the source database recorded in the intermediate database is compared with the metadata of the target database. If there is a difference, the difference result is output. Since the data structure of the source database is inconsistent with the data structure of the target database, the data content cannot be compared, and the entire data comparison process ends; if there is no difference, proceed to the next step.

[0021] Metadata refers to data used to describe data, including but not limited to database information (such as database name and version), data table information (such as table name and creation time), data field information (such as field name and data type), total number of table records, and table unique identifier keys. When the metadata of the source database and the target database exactly match, it indicates that they share the same structural characteristics, allowing the subsequent data comparison process to proceed. Conversely, if the metadata of the source database and the target database differ, it indicates that they represent different entities or structures, and no further comparison is required.

[0022] The intermediate database is a relational database that is used to store detailed information related to each shard, including but not limited to: the shard number (used to identify each shard), the name of the database to which the shard belongs (used to clarify the data source), the name of the data table to which the shard belongs (used to indicate the specific data table location), the shard's unique identification key (as a key field to distinguish different records), the value range of the unique identification key (used to define the coverage of data within the shard), and the hash value calculated using the XXHash algorithm using the unique identification key (used for fast comparison and identification of data consistency).

[0023] (3) Based on the total number of table records, field data types, and other relevant factors in the metadata, determine the shard modulus cardinality size and use the XXHash algorithm to calculate the hash value of the unique identification key; Other relevant factors include: memory occupied by program processing, number of shards, total space occupied by each data entry, total number of records in the table, and space occupied by the table.

[0024] The shard modulo cardinality is calculated using one of the following methods: The modulo cardinality of sharding = (memory occupied by program processing / number of shards) / total space occupied by each piece of data; or, Shard modulus cardinality = (memory occupied by program processing / number of shards) / (table space occupied / total number of records in the table); Choose the best answer based on the situation. If the result is not an integer, round it down.

[0025] The memory used by program processing refers to the memory used by the program while it is running.

[0026] The number of shards: The default is 3, which depends on the number of processor cores.

[0027] The specific operation method for the total space occupied by each piece of data is as follows: 1. Query table structure: obtain the data type and length of each field in the table; 2. Calculate the size of each field: Calculate the size of each field based on the data type and length; 3. Accumulate the size of each field: Add up the size of each field to get the total size of each data item.

[0028] The total number of records in a table refers to the total number of records in the table, which can also be called the total number of rows.

[0029] The space occupied by a table refers to the disk space occupied by a table stored in the database.

[0030] (4) Based on the total number of table records, the shard modulus base, and the hash value in the metadata, determine the number of shards, the shard number, and the hash value range corresponding to each shard. For example, if there are 10 data items, the calculated hash values are 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10. If there are 2 shards in total and the shard modulus base is 5, the hash value range for the first shard is 1-5, and the hash value range for the second shard is 6-10.

[0031] (5) Read the data in the source database and the target database, use the XXHash algorithm to calculate the data of the corresponding shards, generate the corresponding hash value, and record the hash value in the record table of the intermediate database; among them, hash value and hash value are actually different names for the same concept. They both refer to the conversion of input data into an output value of fixed length through a hash function. This output value is usually called a hash value or hash value.

[0032] (6) Based on the shard number, read the data in the source database and the data in the target database, compare the detailed data in the corresponding shards, compare each field of each record one by one, and output the difference results; Specifically: if the hash values of the shards are consistent, it is determined that the data contents of the corresponding shards in the source database and the target database are consistent; if the hash values of the shards are inconsistent, the corresponding detailed data are read from the source database and the target database respectively according to the shard number, database name, data table name, unique identification key and its value range; then, each field of each record in the shard is compared one by one to check whether there are any differences, and detailed difference results are output, including but not limited to field names, field values and their specific differences, so as to determine the specific data inconsistencies.

[0033] The specific implementation examples are as follows: like Figure 2 The following diagram shows the data structure that requires data comparison. The sharding modulus cardinality size is determined based on the total number of tables and field types in the metadata.

[0034] Based on the metadata obtained, the unique key of each table is hashed using XXHash. Combined with the shard modulo cardinality in the previous step, the number of shards required for each table and the unique key range in each shard are obtained. Then, based on the database name, table name, unique key, starting hash value, and ending hash value, the data in the database is read and the hash value is calculated using the XXHash algorithm again, as shown in the following example: Figure 3 shown.

[0035] If the data hash value of a shard is inconsistent, it means that the data on both sides of the shard are inconsistent. Continue to perform a more detailed comparison, such as Figure 4 shown.

[0036] In this method, the total number of table records and the characteristics of field data types significantly impact overall processing efficiency. Therefore, it is necessary to determine the size of the shard modulus base based on these factors to optimize data processing. Furthermore, the use of the non-secure XXHash algorithm not only maintains high computational speed when processing large amounts of data, but also effectively improves overall efficiency by reducing unnecessary computational complexity, enabling efficient and accurate comparison of two or more datasets.

Claims

1. A data comparison method based on metadata, characterized in that: The specific steps include: (1) Obtain the data structure of the source database and the target database, and formulate a corresponding meta-model based on the data structure, where the meta-model is used to describe the metadata attributes of the database and their relationships; (2) Read the metadata of the source database and the target database according to the defined meta-model and database connection information, record the metadata in the intermediate database, compare the metadata in the intermediate database, and output the difference results; (3) Based on the total number of table records, field data types, and other relevant factors in the metadata, determine the shard modulus cardinality size and use the XXHash algorithm to calculate the hash value of the unique identification key; (4) Determine the number of shards, the number of each shard, and the hash value range corresponding to each shard based on the total number of table records, the shard modulus base, and the hash value in the metadata; (5) Read the data in the source database and the target database, use the XXHash algorithm to calculate the data of the corresponding shards, generate the corresponding hash value, and record the hash value in the record table of the intermediate database; (6) Based on the shard number, read the data in the source database and the data in the target database, compare the detailed data in the corresponding shards, compare each field of each record one by one, and output the difference results.

2. The data comparison method based on metadata according to claim 1, characterized in that In step (1), specifically: a meta-model is formulated according to the data structure of the source database, and the meta-model of the source database describes the metadata attributes and their relationships of the source database; another meta-model is formulated according to the data structure of the target database, and the meta-model of the target database describes the metadata attributes and their relationships of the target database.

3. The metadata-based data comparison method according to claim 1 or 2, characterized in that: When the source database type or the target database type is a relational database, a metamodel applicable to the data structure and relational model of the relational database is formulated; when the source database type or the target database type is a non-relational database, a metamodel applicable to the data structure and storage model of the non-relational database is formulated.

4. The data comparison method based on metadata according to claim 1, wherein In step (2), the connection information refers to the information required by the program to access the corresponding database; the intermediate database is an independent database, and the program's persistent data needs to be stored in the intermediate database; specifically: the metadata of the source database and the metadata of the target database are recorded in the intermediate database, and the metadata of the source database recorded in the intermediate database are compared with the metadata of the target database. If there is a difference, the difference result is output. Since the data structure of the source database is inconsistent with the data structure of the target database, the data content cannot be compared, and the entire data comparison process ends; If there is no difference, proceed to the next step.

5. The metadata-based data comparison method according to claim 1 or 4, characterized in that: The metadata refers to data used to describe data, including database information, data table information, data field information, the total number of table records, and the table's unique identification key. When the metadata of the source database completely matches the metadata of the target database, it indicates that the two have the same structural characteristics, and the subsequent data comparison process can continue. Conversely, if there is a difference between the metadata of the source database and the metadata of the target database, it indicates that the two represent different entities or structures, and no further comparison operation is required.

6. The data comparison method based on metadata according to claim 5, characterized in that: The intermediate database is a relational database, which is used to store detailed information related to each shard, including the shard number, the name of the database to which the shard belongs, the name of the data table to which the shard belongs, the unique identification key of the shard, the value range of the unique identification key, and the hash value calculated by the unique identification key using the XXHash algorithm.

7. The data comparison method based on metadata according to claim 1 is characterized in that In step (3), other relevant factors refer to: the memory occupied by program processing, the number of shards, the total space occupied by each piece of data, the total number of records in the table, and the space occupied by the table; the calculation method of the shard modulo cardinality is one of the following methods: shard modulo cardinality = (memory occupied by program processing / number of shards) / total space occupied by each piece of data; or, shard modulo cardinality = (memory occupied by program processing / number of shards) / (space occupied by the table / total number of records in the table); wherein the shard modulo cardinality is rounded down, the memory occupied by program processing refers to the memory occupied by the program during operation, the total number of records in the table refers to the total number of records in the table, and the space occupied by the table refers to the disk space occupied by a table stored in the database.

8. The data comparison method based on metadata according to claim 7 is characterized in that: The specific operation method for the total space occupied by each data is: query the table structure to obtain the data type and length of each field in the table, calculate the size of each field based on the data type and length, and add up the size of each field to get the total size of each data.

9. The data comparison method based on metadata according to claim 1, characterized in that: In step (6), specifically: if the hash values of the shards are consistent, it is determined that the data contents of the corresponding shards in the source database and the target database are consistent; if the hash values of the shards are inconsistent, the corresponding detailed data are read from the source database and the target database respectively according to the shard number, database name, data table name, unique identification key and its value range; then, each field of each record in the shard is compared one by one to check whether there are any differences, and a detailed difference result is output.

Images