Data comparison method

Abstract

The embodiment of the invention provides a data comparison method which comprises the following steps: determining a data matching table and a data conversion table, and determining heterogeneous original data and to-be-compared data. Determining an initial field type corresponding to an original field in the original data in the data matching table, and determining a target field type corresponding to a to-be-compared field in the to-be-compared data in the data matching table; and converting the original field information of the original field into reference field information according to an initial conversion rule corresponding to the initial field type in the data conversion table, and converting the to-be-compared field information of the to-be-compared field into target field information according to a target conversion rule corresponding to the target field type in the data conversion table. The reference field information and the target field information are subjected to data comparison in the same data format dimension to obtain the comparison information, so that the original data and the to-be-compared data are accurately compared, and the data comparison efficiency is improved.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of database technology, and in particular to a data comparison method. Background Art

[0002] During database adaptation and migration, in order to ensure the security and accuracy of data processing, it is usually necessary to compare the data in the two databases for consistency after the migration or adaptation is completed.

[0003] In the prior art, when comparing data from two data sources, to avoid the inefficiency of comparing the entire data set, samples are typically taken from each source and then compared. However, this comparison method cannot guarantee the accuracy of the data comparison. Therefore, a more effective data comparison method is urgently needed to address this issue. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a data comparison method. One or more embodiments of this specification also relate to a data comparison device, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a data comparison method is provided, comprising: Determine the data matching table and data conversion table, as well as the heterogeneous original data and data to be compared; Determining, in the data matching table, an initial field type corresponding to an original field in the original data, and determining, in the data matching table, a target field type corresponding to a to-be-compared field in the to-be-compared data; converting the original field information of the original field into reference field information according to the initial conversion rule corresponding to the initial field type in the data conversion table, and converting the to-be-compared field information of the to-be-compared field into target field information according to the target conversion rule corresponding to the target field type in the data conversion table; Perform data comparison on the reference field information and the target field information to obtain comparison information.

[0006] According to a second aspect of the embodiments of this specification, a data comparison device is provided, comprising: A first determination module is configured to determine a data matching table and a data conversion table, and to determine heterogeneous original data and data to be compared; a second determining module configured to determine, in the data matching table, an initial field type corresponding to an original field in the original data, and to determine, in the data matching table, a target field type corresponding to a to-be-compared field in the to-be-compared data; a conversion module configured to convert the original field information of the original field into reference field information according to an initial conversion rule corresponding to the initial field type in the data conversion table, and to convert the to-be-compared field information of the to-be-compared field into target field information according to a target conversion rule corresponding to the target field type in the data conversion table; The comparison module is configured to perform data comparison on the reference field information and the target field information to obtain comparison information.

[0007] According to a third aspect of an embodiment of this specification, a computing device is provided, including: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned data comparison method are implemented.

[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned data comparison method are implemented.

[0009] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program or instructions, which implement the steps of the above-mentioned data comparison method when executed by a processor.

[0010] The data comparison method provided by one embodiment of the present specification determines a data matching table and a data conversion table, and determines heterogeneous original data and data to be compared. In the data matching table, the initial field type corresponding to the original field in the original data is determined, and in the data matching table, the target field type corresponding to the field to be compared in the data to be compared is determined. The data matching table is queried for the original field and the field to be compared respectively to determine the initial field type and the target field type, thereby determining the data type. According to the initial conversion rule corresponding to the initial field type in the data conversion table, the original field information of the original field is converted into reference field information, and according to the target conversion rule corresponding to the target field type in the data conversion table, the field information to be compared of the field to be compared is converted into target field information. By converting the original field information and the field information to be compared respectively, the original field information and the field information to be compared are serialized and processed into reference field information and target field information of the same data format. The reference field information and the target field information are compared in the same data format dimension to obtain comparison information, thereby achieving accurate comparison of the original data and the data to be compared, thereby improving data comparison efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a schematic diagram of a data comparison method provided in one embodiment of this specification; Figure 2 This is a flow chart of a data comparison method provided by one embodiment of this specification; Figure 3 This is a flow chart of a processing process of a data comparison method provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the structure of a data comparison device provided in one embodiment of this specification; Figure 5 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0012] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0013] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0014] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0015] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0016] First, the terms involved in one or more embodiments of this specification are explained.

[0017] UUID: The full name is Universally Unique Identifier, which is a standard 16-byte (128-bit) long digital identifier.

[0018] enum: An enumeration is a user-defined data type that consists of a set of named constants. Enumerations are supported in many programming languages, including C, C++, Java, and C#, and provide an efficient way to define a set of constants with the same meaning.

[0019] Boolean: Boolean is a data type used to represent logical true or false.

[0020] During database adaptation and migration, developers often need to compare the consistency of data processing results between two databases after the migration or adaptation is complete. When comparing heterogeneous data, differences in database implementation, inconsistent data type implementations, and inconsistent supported data types can result in data with the same business meaning being represented differently in different databases or database driver implementations, making direct comparison impossible. Data comparison methods such as sampling partial fields for comparison or using customized scripts cannot guarantee the accuracy of the comparison results. Therefore, a more effective data comparison method is urgently needed to address these issues. Figure 1 A schematic diagram of a data comparison method provided according to an embodiment of the present specification is shown. When performing heterogeneous data comparison, a data matching table and a data conversion table can be determined, as well as heterogeneous original data and data to be compared.

[0021] Furthermore, the initial field type corresponding to the original field in the original data can be determined in the data matching table, and the target field type corresponding to the field to be compared in the data to be compared can be determined in the data matching table. The data matching table can be queried for the original field and the field to be compared respectively to determine the initial field type and the target field type to determine the data type. According to the initial conversion rule corresponding to the initial field type in the data conversion table, the original field information of the original field is converted into reference field information, and according to the target conversion rule corresponding to the target field type in the data conversion table, the field information to be compared of the field to be compared is converted into target field information. By converting the original field information and the field information to be compared respectively, the original field information and the field information to be compared are serialized and processed into reference field information and target field information of the same data format. The reference field information and the target field information are compared in the same data format dimension to obtain comparison information, thereby achieving accurate comparison between the original data and the data to be compared and improving data comparison efficiency.

[0022] In this specification, a data comparison method is provided. This specification also relates to a data comparison device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0023] See also Figure 2 , Figure 2 A flow chart of a data comparison method provided according to an embodiment of this specification is shown, which specifically includes the following steps.

[0024] Step 202: Determine a data matching table and a data conversion table, and determine heterogeneous original data and data to be compared.

[0025] Specifically, heterogeneous raw data and data to be compared correspond to different data sources. In a data migration scenario, the raw data in the original data source needs to be migrated to the target data source. After the raw data is migrated to the target data source, the data to be compared can be obtained from the target data source. This means that after the raw data is migrated to the target data source, it needs to be compared with the raw data stored in the original data source to determine whether the data consistency between the original and target data sources is maintained. A data conversion table is used to store at least one data conversion rule. Based on the data conversion rules stored in the data conversion table, the representation of the raw data and the data to be compared can be converted. Each data conversion rule in the data conversion table corresponds to a different type of domain data. When determining data conversion rules for the raw data and the data to be compared, the applicable data conversion rule can be selected based on the domain data types corresponding to the raw data and the data to be compared. The data matching table records field names and the domain data types corresponding to the field names. The data matching table is used to determine the domain data types corresponding to the fields contained in the raw data and the data to be compared, respectively.

[0026] Based on this, before comparing heterogeneous raw data and data to be compared, you can first determine the data matching table and data conversion table, as well as the heterogeneous raw data and data to be compared. Subsequently, you can convert the data representation of the raw data and data to be compared in the field dimension based on the data matching table and data conversion table before performing the data comparison.

[0027] In practical applications, the data comparison method provided in this embodiment can also be applied to data adaptation scenarios. In data adaptation scenarios, after data adaptation is completed, data comparison is also required to ensure data consistency.

[0028] Furthermore, the data conversion table can be generated by defining the priority and data conversion rules corresponding to the domain data type. The specific implementation is as follows: Determine at least one type of domain data, and determine the data conversion rules and priorities corresponding to the at least one type of domain data; construct the data conversion table based on the data conversion rules and priorities corresponding to the at least one type of domain data.

[0029] Specifically, the domain data type refers to the business type determined based on the data type corresponding to the original data source of the original data. The data types include but are not limited to date, time, system time, Boolean, enumeration, number, byte, string, uuid, sequence, random number and other data types. Correspondingly, the business types include but are not limited to uuid, random number, sequence, system time, date / time / timestamp / datetime, boolean, enum, number / decimal, bytea / blob, char / varchar / text / clob, and other data types. Different priority levels can be specified for each type of business. The data conversion rule represents the serialization method for serializing the fields in the original data and the fields in the data to be compared, that is, the field values are serialized and processed into the same serialization format.

[0030] Based on this, at least one domain data type is determined based on the data types corresponding to the original data source of the original data and the target data source of the data to be compared, and data conversion rules and priorities corresponding to the at least one domain data type are determined. A data conversion table is constructed based on the data conversion rules and priorities corresponding to the at least one domain data type.

[0031] For example, before comparing the original data from the original data source and the data to be compared from the target data source, at least one domain data type can be set based on the data types associated with the original data source and the target data source of the data to be compared, and a priority can be assigned to each domain data type to construct a data conversion table. The determined domain data types include but are not limited to uuid, random number, sequence, system time, date / time / timestamp / datetime, boolean, enum, number / decimal, bytea / blob, char / varchar / text / clob, and all data types. All data types can be varchar(100), clob, decimal(10,2), char(1), etc.

[0032] In summary, by defining the priorities and data conversion rules corresponding to domain data types and generating a data conversion table, the serialization method can be customized, providing a unified data serialization method for subsequent data comparison.

[0033] Step 204: Determine in the data matching table an initial field type corresponding to an original field in the original data, and determine in the data matching table a target field type corresponding to a to-be-compared field in the to-be-compared data.

[0034] Specifically, after determining the data matching table and data conversion table, and determining the heterogeneous raw data and data to be compared, the data matching table can determine the initial field type corresponding to the original field in the raw data, and the data matching table can determine the target field type corresponding to the to-be-compared field in the to-be-compared data. The raw data is stored in the form of key-value pairs, containing at least one key-value pair, each of which includes a field name as a key and field information as a value. Accordingly, the data to be compared is also stored in the form of key-value pairs, with the number of key-value pairs in the to-be-compared data being the same as the number of key-value pairs in the raw data. The initial field type corresponding to the original field refers to the field business type corresponding to the original matching field recorded in the data matching table after the original matching field matching the original field is determined in the data matching table based on the raw data's original field. Accordingly, the target field type corresponding to the to-be-compared field refers to the target field type corresponding to the to-be-compared field recorded in the data matching table after the to-be-compared field matching the to-be-compared field is determined in the data matching table based on the to-be-compared field in the to-be-compared data.

[0035] Based on this, after determining the data matching table and data conversion table, and determining the heterogeneous original data and data to be compared, the original fields are determined in the original data, and the original matching fields corresponding to the original fields in the original data are determined in the data matching table. Furthermore, the initial field types corresponding to the original matching fields are determined in the data matching table. The fields to be compared are determined in the data to be compared, and the matching fields to be compared corresponding to the fields to be compared in the data to be compared are determined in the data matching table. Furthermore, the target field types corresponding to the matching fields to be compared are determined in the data matching table.

[0036] Furthermore, the data matching table records the field names and the domain data types corresponding to the field names. By querying the data matching table based on the original fields, the initial field type corresponding to the original fields can be determined. The specific implementation is as follows: The original field is determined in the original data, and the data matching table is queried based on the original field; when it is determined according to the original query result that the data matching table contains an initial record matching the original field, the initial field type corresponding to the original field is determined based on the initial record; when it is determined according to the original query result that the data matching table does not contain a record matching the original field, the original data type corresponding to the original field is determined according to the original data, and the initial field type corresponding to the original field is determined based on the original data type.

[0037] Specifically, the original query result refers to the query result determined by querying the data matching table based on the original field. The original query result can be the initial record found in the data matching table that matches the original field, or it can be the result of no record found in the data matching table that matches the original field. The initial record that matches the original field contains the field name of the original field.

[0038] Based on this, the original field is determined in the original data, and the data matching table is queried based on the field information (field name) corresponding to the original field. If the data matching table is determined to contain an initial record that matches the original field based on the original query results, this indicates that the data matching table contains field information identical to the field information of the original field, and the field data type corresponding to the field information in the initial record is used as the initial field type corresponding to the original field. If the data matching table is determined to not contain a record that matches the original field based on the original query results, this indicates that the data matching table does not contain field information identical to the field information of the original field, that is, the initial field type corresponding to the original field cannot be determined based on the data matching table. In this case, the data type of the original data recorded in the original data source (original database) can be used as the original data type corresponding to the original field, and the data conversion table can be searched based on the original data type to determine the initial field type of the corresponding original field that matches the original data type in the data conversion table.

[0039] Continuing with the above example, the data matching table records multiple field names and the domain data type corresponding to each field name. The data matching table is shown in Table 1 below:

[0040] Table 1 When the original field is userId, the data matching table is queried based on the original field userId to determine the initial record "{"tableName":"users","columnName":["userId"]}-sequence". Based on the initial record, it can be determined that the initial field type corresponding to the original field userId is a sequence type. When the original field is userName, the data matching table is queried based on the original field userName. If no record corresponding to the original field userName is found, the data type of the original field userName recorded in the original database is determined to be varchar(100). Based on the data type varchar(100), the data conversion table is queried to determine all data types corresponding to the original field userName.

[0041] In summary, the data matching table is queried based on the original field, and the original data type corresponding to the original field is determined based on the original query results. If the data matching table does not contain a record matching the original field, the initial field type of the corresponding original field is determined based on the data conversion table, thereby ensuring that each original field in the original data has a corresponding initial field type.

[0042] Furthermore, considering that the data matching table records multiple field names and the domain data type corresponding to each field name, some of the multiple field names contain data source identifiers. When querying the data matching table based on the original fields, you can first query and match the fields for some field names containing data source identifiers. The specific implementation is as follows: Determine the original field identifier corresponding to the original field, and query the data matching table based on the original field identifier and the original field; if it is determined based on the original identifier query result that the data matching table contains the initial record that matches the original field, use the initial record as the original query result; if it is determined based on the original identifier query result that the data matching table does not contain a record that matches the original field, query the data matching table based on the original field.

[0043] Specifically, the original field identifier is the data source identifier. The original field identifier can be a database name (database identifier) or a table name (table identifier). The original identifier query result is the query result determined based on the original field identifier and the original field query data matching table.

[0044] Based on this, the database identifier or data table identifier corresponding to the original field is used as the original field identifier, and the data matching table is queried based on the original field identifier and the original field. The field identifier record corresponding to the original field identifier is first determined in the data matching table, and the initial record is determined in the field identifier record based on the original field. When it is determined based on the original identifier query result that the data matching table contains the initial record that matches the original field, it means that the data matching table contains the initial record that matches both the original field identifier and the original field, and the initial record is used as the original query result. When it is determined based on the original identifier query result that the data matching table does not contain the record that matches the original field, it means that the data matching table does not contain the initial record that matches both the original field identifier and the original field, and the data matching table is queried again based on the original field.

[0045] The original data is "{"userId":{"dataType":"int","value":123},"userName":{"dataType":"varchar","value":"xyz "},"key":{"dataType":"varchar","value":"550e8400-e29b-41d4-a716-446655440000"},"status":{"da taType":"boolean","value":true},"amount":{"dataType":"number","value":12.10},"createDate":{"dataType":"date","value":"2021-12-12T12:32:32"},"label":{"dataType":"varchar","value":""}}". If the original fields in the original data are userId, userName, key, status, amount, createDate, label, and the original field identifier of the original data is t1, you can query Table 1 above based on each field to determine the domain data type (initial field type) corresponding to each field. If the original field is key, you can first query the data matching table based on the original field identifier t1 and the original field key. First, query the data matching table based on the original field identifier t1. If the data matching table does not contain the original field identifier t1 and the record that matches the original field key, directly query the data matching table based on the original field key to determine that the initial field type corresponding to the original field key is uuid. Similarly, you can determine the target field type corresponding to each original field, that is, userId corresponds to sequence; userName corresponds to all data types; key corresponds to uuid; status corresponds to boolean; amount corresponds to all data types; createDate corresponds to system time; label corresponds to all data types.

[0046] In summary, querying the data matching table based on the original field identifier and the original field can improve the efficiency of determining the initial field type corresponding to the original field.

[0047] Furthermore, the data matching table records the field names and the domain data types corresponding to the field names. By querying the data matching table based on the field to be compared, the target field type corresponding to the field to be compared can be determined. The specific implementation is as follows: The field to be compared is determined in the data to be compared, and the data matching table is queried based on the field to be compared; when it is determined according to the query result to be compared that the data matching table contains a target record that matches the field to be compared, the target field type corresponding to the field to be compared is determined based on the target record; when it is determined according to the query result to be compared that the data matching table does not contain a record that matches the field to be compared, the data type to be compared corresponding to the field to be compared is determined according to the data to be compared, and the target field type corresponding to the field to be compared is determined according to the data type to be compared.

[0048] Specifically, the query result to be compared refers to the query result determined by querying the data matching table based on the field to be compared. The query result to be compared can be a target record that matches the field to be compared in the data matching table, or it can be a record that does not match the field to be compared in the data matching table. The target record that matches the field to be compared contains the field name of the field to be compared.

[0049] Based on this, a field to be compared is determined in the data to be compared, and a data matching table is queried based on the field information (field name) corresponding to the field to be compared. If the query results determine that the data matching table contains a target record that matches the field to be compared, this indicates that the data matching table contains field information identical to the field information of the field to be compared, and the field data type corresponding to the field information to be compared in the target record is used as the target field type corresponding to the field to be compared. If the query results determine that the data matching table does not contain a record that matches the field to be compared, this indicates that the data matching table does not contain field information identical to the field information of the field to be compared, i.e., the target field type corresponding to the field to be compared cannot be determined based on the data matching table. In this case, the data type of the data recorded in the target data source (target database) of the data to be compared can be used as the data type corresponding to the field to be compared. Based on the data type to be compared, a data conversion table is searched and a target field type corresponding to the field to be compared that matches the data type to be compared is determined in the data conversion table.

[0050] When the field to be compared is userId, the data matching table is queried based on the field to be compared userId to determine the target record "{"tableName":"users","columnName":["userId"]}-sequence". Based on the target record, it can be determined that the target field type corresponding to the field to be compared userId is a sequence type. When the field to be compared is userName, the data matching table is queried based on the field to be compared userName. If no record corresponding to the field to be compared userName is found, the data type of the field to be compared userName recorded in the target database is determined to be varchar(100). Based on the data type varchar(100), the data conversion table is queried to determine that the field to be compared userName corresponds to all data types.

[0051] In summary, the data matching table is queried based on the field to be compared, and the target data type corresponding to the field to be compared is determined based on the query results. If the query results determine that the data matching table does not contain a record matching the field to be compared, the target field type corresponding to the field to be compared is determined based on the data conversion table, thereby ensuring that each field to be compared in the data to be compared has a corresponding target field type.

[0052] Furthermore, considering that the data matching table records multiple field names and the domain data type corresponding to each field name, and some of the multiple field names contain data source identifiers, when querying the data matching table based on the field to be compared, you can first query and match the fields for some of the field names containing the data source identifier. The specific implementation is as follows: Determine a field identifier to be compared corresponding to the field to be compared, and query the data matching table based on the field identifier to be compared and the field to be compared; if it is determined based on the query result of the identifier to be compared that the data matching table contains the target record that matches the field to be compared, use the target record as the query result to be compared; if it is determined based on the query result of the identifier to be compared that the data matching table does not contain a record that matches the field to be compared, query the data matching table based on the field to be compared.

[0053] Specifically, the field identifier to be compared is the data source identifier. The field identifier to be compared can be the database name (database identifier) or table name (table identifier) corresponding to the data to be compared. The query result for the identifier to be compared is the query result determined based on the field identifier to be compared and the matching table for the data query of the field to be compared.

[0054] Based on this, the database identifier or data table identifier corresponding to the field to be compared is used as the field identifier to be compared, and the data matching table is queried based on the field identifier to be compared and the field to be compared. The field identifier record to be compared corresponding to the field identifier to be compared is first determined in the data matching table, and the target record is determined in the field identifier record to be compared based on the field to be compared. When it is determined based on the query result of the identifier to be compared that the data matching table contains the target record that matches the field to be compared, it means that the data matching table contains the target record that matches both the field identifier to be compared and the field to be compared, and the target record is used as the query result to be compared. When it is determined based on the query result of the identifier to be compared that the data matching table does not contain the record that matches the field to be compared, it means that the data matching table does not contain the target record that matches both the field identifier to be compared and the field to be compared, and the data matching table is queried again based on the field to be compared.

[0055] The data to be compared is: "{"userId":{"dataType":"int","value":123},"userName":{"dataType":"varchar","value": "xyz"},"key":{"dataType":"varchar","value":"09F686761827CF8AE040578CB20B7491"},"status":{"d ataType":"number","value":1},"amount":{"dataType":"number","value":12.1},"createDate":{"dataType":"date","value":"2021-12-1212:32:34"},"label":{"dataType":"varchar","value":null}}". If the fields to be compared in the data to be compared are userId, userName, key, status, amount, createDate, label, and the field identifier is t2, Table 1 can be queried for each field to determine the corresponding domain data type (target field type). If the field to be compared is key, the data matching table can be queried based on the field identifier t2 and the field key. If no record in the data matching table contains the field identifier t2 and matches the field key, the data matching table can be directly queried based on the field key to determine that the target field type corresponding to the field key is uuid. Similarly, the target field type corresponding to each original field can be determined: userId corresponds to sequence; userName corresponds to all data types; key corresponds to uuid; status corresponds to all data types; amount corresponds to all data types; createDate corresponds to system time; and label corresponds to all data types.

[0056] In summary, querying the data matching table based on the to-be-compared field identifier and the to-be-compared field can improve the efficiency of determining the target field type corresponding to the to-be-compared field.

[0057] Step 206: According to the initial conversion rule corresponding to the initial field type in the data conversion table, the original field information of the original field is converted into reference field information, and according to the target conversion rule corresponding to the target field type in the data conversion table, the to-be-compared field information of the to-be-compared field is converted into target field information.

[0058] Specifically, after determining the initial field type corresponding to the original field in the original data in the data matching table and the target field type corresponding to the to-be-compared field in the to-be-compared data in the data matching table, the original field information of the original field can be converted into reference field information according to the initial conversion rule corresponding to the initial field type in the data conversion table, and the to-be-compared field information of the to-be-compared field can be converted into target field information according to the target conversion rule corresponding to the target field type in the data conversion table. The initial conversion rule corresponding to the initial field type is a pre-set serialization rule for serializing the original field information of the original field in the original data. Original data is stored in the form of key-value pairs, and accordingly, original fields are also stored in the form of key-value pairs, with the field name of the original field being the key and the original field information of the original field being the value. For example, the original field "userId" is: {"dataType":"int","value":123}, where userId is the key and {"dataType":"int","value":123} is the value, separated by a semicolon (";"). Reference field information is the information obtained by converting the original field information according to the initial conversion rules. The target conversion rules corresponding to the target field type are pre-defined serialization rules used to serialize the field information of the comparison field in the comparison data. The comparison data is stored as key-value pairs, and accordingly, the comparison fields are also stored as key-value pairs, with the field name of the comparison field as the key and the field information of the comparison field as the value. The target field information is the information obtained by converting the field information according to the target conversion rules.

[0059] Based on this, after determining the initial field type corresponding to the original field in the original data in the data matching table, and determining the target field type corresponding to the field to be compared in the data to be compared in the data matching table, the original field information of the original field is converted into reference field information according to the initial conversion rule corresponding to the initial field type in the data conversion table, thereby realizing serialization processing of the original field information. According to the target conversion rule corresponding to the target field type in the data conversion table, the field information to be compared of the field to be compared is converted into target field information, thereby realizing serialization processing of the field information to be compared. This facilitates subsequent data comparison based on the original field information and the field information to be compared after serialization processing. The original field information and the field information to be compared after serialization processing have the same data format, which facilitates data comparison.

[0060] Furthermore, the initial field type can be one of multiple domain data types recorded in the data conversion table, which can be determined by directly looking up the domain data type. The initial field type can also be a domain data subtype contained in a domain data type in the data conversion table. This needs to be determined in combination with the priority in the data conversion table. The specific implementation is as follows: In the case where the initial field type is the initial business type, the initial conversion rule is determined in the data conversion table based on the initial business type; in the case where the initial field type is the initial data type, the initial conversion rule corresponding to the initial business type is searched in the data conversion table in order of priority; the original field information corresponding to the original field is determined, and the original field information is converted according to the initial conversion rule to obtain the reference field information.

[0061] Specifically, the initial business type is the data type of the domain business type in the corresponding data conversion table. The initial data type is the field type of the original field obtained from the database.

[0062] Based on this, when the initial field type is the initial business type, it means that the original field has been assigned a data type. The initial conversion rule can be directly based on the domain data type corresponding to the initial business type in the data conversion table and the serialization method corresponding to the determined domain data type. When the initial field type is the initial data type, it means that the original field has not been assigned a data type. The initial data type corresponding to the original field recorded in the database can be used to search the data conversion table for the initial conversion rule corresponding to the initial business type in order of priority. The original field information corresponding to the original field is determined, and the original field information is converted according to the initial conversion rule to obtain the reference field information.

[0063] Continuing with the above example, for the original fields in the original data: userId, userName, key, status, amount, createDate, label. After determining the initial field type of each original field, perform field information conversion processing. For the original field userId, since the data matching table contains an initial business type (sequence) corresponding to userId, the initial conversion rule can be determined according to the serialization method corresponding to the initial business type (sequence) in the data conversion table. Based on the initial conversion rule, the original field information "{"dataType":"int","value":123}" corresponding to the original field userId is converted to obtain the reference field information 123. For the original field userName, since there is no corresponding business type in the data matching table, the data type varchar(100) recorded in the database is used as the initial business type. The initial conversion rule is determined according to the serialization method corresponding to the initial business type varchar(100) in the data conversion table. Based on the initial conversion rule, the original field information "{"dataType":"varchar","value":"xyz"}" corresponding to the original field userName is converted to obtain the reference field information xyz. This process is repeated until the reference field information corresponding to each original field is obtained.

[0064] In summary, an initial conversion rule matching the original field is determined according to the initial field type of the original field, and then the original field information corresponding to the original field is converted according to the determined initial conversion rule, thereby improving the accuracy of field information conversion.

[0065] Furthermore, the target field type can be one of multiple domain data types recorded in the data conversion table, which can be determined by directly searching the domain data type. The target field type can also be a domain data subtype contained in a domain data type in the data conversion table. It needs to be determined in combination with the priority in the data conversion table. The specific implementation is as follows: In the case where the target field type is a target business type, the target conversion rule is determined in the data conversion table based on the target business type; in the case where the target field type is a target data type, the target conversion rule corresponding to the target business type is searched in the data conversion table in order of priority; the field information to be compared corresponding to the field to be compared is determined, and the field information to be compared is converted according to the target conversion rule to obtain the target field information.

[0066] Specifically, the target business type is the data type of the domain business type in the corresponding data conversion table. The target data type is the field type of the field to be compared obtained from the database.

[0067] Based on this, when the target field type is the target business type, it means that the field to be compared has been assigned a data type. The target conversion rule can be directly based on the field data type corresponding to the target business type in the data conversion table and the serialization method corresponding to the determined field data type. When the target field type is the target data type, it means that the original field has not been assigned a data type. The target data type corresponding to the field to be compared recorded in the database can be used to search the target conversion rule corresponding to the target business type in the data conversion table in order of priority. The field information to be compared corresponding to the field to be compared is determined, and the field information to be compared is converted according to the target conversion rule to obtain the target field information.

[0068] Continuing with the above example, for the fields to be compared in the data to be compared: userId, userName, key, status, amount, createDate, label. After determining the target field type for each field to be compared, perform field information conversion. For the field to be compared, userId, since the data matching table contains a target business type (sequence) corresponding to userId, the target conversion rule can be determined according to the serialization method corresponding to the target business type (sequence) in the data conversion table. Based on the target conversion rule, the field information to be compared, "{"dataType":"int","value":123}" corresponding to the field to be compared, is converted to obtain the target field information 123. For the field to be compared, userName, since there is no corresponding business type in the data matching table, the data type varchar(100) recorded in the database is used as the target business type. The target conversion rule is determined according to the serialization method corresponding to the target business type varchar(100) in the data conversion table. Based on the target conversion rule, the field information to be compared, "{"dataType":"varchar","value":"xyz"}" corresponding to the field to be compared, userName, is converted to obtain the target field information xyz. This process is repeated until the target field information corresponding to each field to be compared is obtained.

[0069] In summary, the target conversion rule matching the field to be compared is determined according to the target field type of the field to be compared, and then the field information to be compared corresponding to the field to be compared is converted according to the determined target conversion rule, thereby improving the accuracy of field information conversion.

[0070] Furthermore, considering that a plurality of serialization modes as conversion rules are provided in the data conversion table, when two or more conversion rules can be applied to serialize the same field information, the priorities corresponding to the two or more conversion rules can be reset to update the rule priorities in the data conversion table and adjust the order of application of the conversion rules. The specific implementation is as follows: When it is determined that the initial field type contains at least two types of initial field subtypes, the data conversion table is updated with rule priorities according to the initial field conversion information corresponding to the at least two types of initial field subtypes; when it is determined that the target field type contains at least two types of target field subtypes, the data conversion table is updated with rule priorities according to the target field conversion information corresponding to the at least two types of target field subtypes.

[0071] Based on this, if it is determined that the initial field type contains at least two initial field subtypes, this indicates that when performing field type matching based on the original field in the data matching table, at least two initial field types were matched. These at least two matched initial field types are at least two initial field subtypes. The data conversion table is updated with the rule priorities based on the initial field conversion information corresponding to the at least two initial field subtypes. The initial field conversion information may be conversion rule adjustment information determined by a user using the data comparison function, used to adjust the conversion rules in the data conversion table based on the priority dimension under the influence of the initial field type. If it is determined that the target field type contains at least two target field subtypes, this indicates that when performing field type matching based on the to-be-compared field in the data matching table, at least two target field types were matched. These at least two matched target field types are at least two target field subtypes. The data conversion table is updated with the target field conversion information corresponding to the at least two target field subtypes. The target field conversion information may be conversion rule adjustment information determined by a user using the data comparison function, used to adjust the conversion rules in the data conversion table based on the priority dimension under the influence of the target field type.

[0072] Using the above example, if the field name is mask and the conversion rules for both UUID and all data types are used, since UUID corresponds to a 32-bit string containing "-", applying the conversion rule for UUID first, followed by the conversion rule for all data types, will yield the same conversion results. However, applying the conversion rule for all data types first may result in two conversion results: one containing "abc-" and the other not containing "abc-", causing data comparison failure. In this case, you can adjust the order in which the conversion rules are applied by adjusting their priority.

[0073] In summary, the data conversion table is updated with rule priorities based on target field conversion information corresponding to at least two target field subtypes. This allows for the purpose of adjusting the order in which conversion rules are applied within the data conversion table, enabling users to fine-tune the order in which conversion rules are applied, and improving the flexibility of data comparison.

[0074] Step 208: Perform data comparison on the reference field information and the target field information to obtain comparison information.

[0075] Specifically, after converting the original field information of the original field into reference field information according to the initial conversion rules corresponding to the initial field type in the data conversion table, and converting the to-be-compared field information of the to-be-compared field into target field information according to the target conversion rules corresponding to the target field type in the data conversion table, data comparison can be performed on the reference field information and the target field information to obtain comparison information, wherein the goal of performing data comparison on the reference field information and the target field information is to perform field value comparison, that is, to perform an accurate comparison according to the string format of the reference field information and the target field information. The comparison information can include comparison information for comparing the reference field information and the target field information one by one, as well as comparison results indicating whether the data comparison is identical or different.

[0076] Based on this, after converting the original field information of the original field into reference field information according to the initial conversion rules corresponding to the initial field type in the data conversion table, and converting the field information to be compared into target field information according to the target conversion rules corresponding to the target field type in the data conversion table, the reference field information and the target field information are compared field by field to obtain comparison information.

[0077] Furthermore, considering that the reference field information includes at least one reference field value and the target field information includes at least one target field value, a field-by-field comparison is required when comparing the reference field information and the target field information. The specific implementation is as follows: Determine at least one reference field value contained in the reference field information, and determine at least one target field value contained in the target field information; construct a field value data pair based on the at least one reference field value and the at least one target field value, and perform a consistency comparison on the field value data pair to obtain the comparison information.

[0078] Specifically, reference field information includes at least one reference field value, each corresponding to a different reference field name; the reference field name and reference field value form a reference key-value pair; target field information includes at least one target field value, each corresponding to a different target field name; the target field name and target field value form a target key-value pair. A field value data pair consists of a reference key-value pair and a target key-value pair with the same field name. Within a field value data pair, the reference field value in the reference key-value pair and the target field value in the target key-value pair need to be compared.

[0079] Based on this, at least one reference field value included in the reference field information is determined, and at least one target field value included in the target field information is determined. Based on the at least one reference field value and the at least one target field value, a field value data pair associated with the same field name is constructed, and a consistency comparison is performed on the reference field value and the target field value in the field value data pair to obtain comparison information.

[0080] Continuing with the above example, the reference field information contains multiple reference field values, each corresponding to a different reference field name. The reference field names include userId, userName, key, status, amount, createDate, and label. Correspondingly, the target field names also include userId, userName, key, status, amount, createDate, and label. The reference field names correspond one-to-one to the reference field values: ""userId"-123,"userName"-xyz,"key"-uuid,"status"-true,"amount"-12.1,"createDate"-system time, and "label"-null. Correspondingly, the target field names correspond one-to-one to the target field values: ""userId"-123,"userName"-xyz,"key"-uuid,"status"-true,"amount"-12.1,"createDate"-system time, and "label"-null. A data comparison table can be constructed based on the above information to facilitate data comparison field by field. The following Table 2 is constructed based on the above information:

[0081] Table 2 In summary, t1 corresponds to the reference field information, and t2 corresponds to the target field information. Based on Table 2 above, the reference field information and the target field information can be compared field by field to obtain comparison information. Finally, it is determined that the comparison results of the reference field information and the target field information are consistent.

[0082] Furthermore, considering that the comparison between the reference field information and the target field information needs to be performed field by field, if one field fails the comparison, it is considered that the reference field information and the target field information are different and the comparison fails. In this case, the target field information that fails the comparison can be marked with a rule to facilitate rule analysis. The specific implementation is as follows: When it is determined according to the comparison information that the original data and the data to be compared do not pass the data comparison, a target field information pair is determined in the reference field information and the target field information; and the target field information pair is rule-marked in the metadata dimension.

[0083] Specifically, a target field information pair includes the reference field name and the target field name, as well as the reference field value corresponding to the reference field name and the target field value corresponding to the target field name. Inconsistencies between the reference field value and the target field value will cause the original data and the data to be compared to fail the data comparison. Rule marking refers to the metadata marking of the target transformation rule applied to the target field value and the initial transformation rule applied to the reference field value.

[0084] Based on this, if the original data and the data to be compared are determined to have failed data comparison based on the comparison information, this indicates that at least one set of different field values exists when comparing the multiple reference field values corresponding to the original data and the multiple target field values corresponding to the data to be compared. Target field information pairs with different field values are identified in the reference field information and the target field information. Historical conversion rules used during data conversion for the target field information pairs are determined, and metadata dimensions are marked with the rules, with the historical conversion rules annotated in the metadata.

[0085] Continuing with the previous example, if the reference field name and value corresponding to the original data are userId and 123, respectively, and the target field name and value corresponding to the data to be compared are userId and 1234, respectively, since the reference field values are inconsistent with the target field values, the target field information pair is determined to be userId-123:userId-1234. Determine the conversion rule to use for the target field value and the conversion rule to use for the reference field value, and mark the conversion rule used in the metadata.

[0086] To summarize, when it is determined based on the comparison information that the original data and the data to be compared have not passed the data comparison, the historical conversion rules used for the target field information during data conversion are determined, and the metadata dimensions are marked with rules. The historical conversion rules are annotated in the metadata to facilitate subsequent comparison result analysis.

[0087] The data comparison method provided by one embodiment of the present specification determines a data matching table and a data conversion table, and determines heterogeneous original data and data to be compared. In the data matching table, the initial field type corresponding to the original field in the original data is determined, and in the data matching table, the target field type corresponding to the field to be compared in the data to be compared is determined. The data matching table is queried for the original field and the field to be compared respectively to determine the initial field type and the target field type, thereby determining the data type. According to the initial conversion rule corresponding to the initial field type in the data conversion table, the original field information of the original field is converted into reference field information, and according to the target conversion rule corresponding to the target field type in the data conversion table, the field information to be compared of the field to be compared is converted into target field information. By converting the original field information and the field information to be compared respectively, the original field information and the field information to be compared are serialized and processed into reference field information and target field information of the same data format. The reference field information and the target field information are compared in the same data format dimension to obtain comparison information, thereby achieving accurate comparison of the original data and the data to be compared, thereby improving data comparison efficiency.

[0088] The following combined Figure 3 , taking the application of the data comparison method provided in this specification in the data migration scenario as an example, the data comparison method is further explained. Figure 3 A processing flow chart of a data comparison method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0089] Step 302: Determine at least one type of domain data, and determine the data conversion rules and priorities corresponding to the at least one type of domain data, and construct a data conversion table based on the data conversion rules and priorities corresponding to the at least one type of domain data.

[0090] In data migration scenarios (from a source database to a target database), after completing the migration, it is often necessary to compare the data in the two databases to ensure consistency. Due to implementation differences and inconsistent data types between databases, direct data comparison is not possible. Therefore, this embodiment proposes a data comparison method that serializes the original data in the source database and the data to be compared in the target database according to a predefined serialization method before performing the data comparison, thereby improving the accuracy and efficiency of the data comparison.

[0091] In actual applications, domain data types are commonly used business domain types, including but not limited to numbers, dates, times, Booleans, enumerations, strings, bytes, UUIDs, system time, sequences, random numbers and other data types. The priority of each domain data type can be set according to actual business needs, and this embodiment does not impose too many restrictions on this.

[0092] Step 304: Determine heterogeneous original data and data to be compared.

[0093] The source database acts as the data migration source and provides the original data, while the target database acts as the data migration destination and provides the data to be compared.

[0094] Step 306: Determine the original field in the original data, and query the data matching table based on the original field.

[0095] The data matching table is used to determine the domain data type corresponding to each field in the original data. The data matching table records the field information and the domain data type corresponding to the field information.

[0096] Step 308: When it is determined according to the original query result that the data matching table contains an initial record matching the original field, determine the initial field type corresponding to the original field based on the initial record.

[0097] The original data contains multiple original fields, and the data matching table needs to be queried based on each original field in turn. In the case of the original data corresponding table identifier (original field identifier) T1, the data matching table can be queried for the first time based on the original field identifier and the original field. If a record containing T1 that matches the original field is found, the field data type in the record can be used as the initial field type corresponding to the original field. If a record containing T1 that matches the original field is not found, the data matching table can be queried for the second time based on the original field to obtain the initial field type corresponding to the original field.

[0098] Step 310: When it is determined according to the original query result that the data matching table does not contain a record matching the original field, the original data type corresponding to the original field is determined according to the original data, and the initial field type corresponding to the original field is determined according to the original data type.

[0099] If the original field type corresponding to the original field is not obtained by querying the data matching table a second time based on the original field, the original field type corresponding to the original field can be determined based on the original data type obtained from the source database. Subsequently, the data conversion table can be queried based on the priority recorded in the data conversion table to determine the serialization method that can serialize the original field information corresponding to the original field.

[0100] Step 312: When the initial field type is the initial business type, determine an initial conversion rule in the data conversion table based on the initial business type.

[0101] When the initial field type is the initial business type determined by querying the data matching table, the data conversion table can be directly queried based on the initial business type to determine the initial conversion rule that can serialize the original field information corresponding to the original field.

[0102] Step 314: When the initial field type is the initial data type, search the data conversion table for the initial conversion rule corresponding to the initial business type in order of priority.

[0103] When the initial field type is the initial data type (original data type) obtained from the source database, the data conversion table can be queried in combination with the priority recorded in the data conversion table to determine a serialization method that can serialize the original field information corresponding to the original field, and use the serialization method as the initial conversion rule.

[0104] Step 316: Determine the original field information corresponding to the original field, and convert the original field information according to the initial conversion rule to obtain reference field information.

[0105] The data in the original data may be stored in the form of key-value pairs, and the value of the original field may be serialized as original field information to obtain reference field information.

[0106] Step 318: Determine the fields to be compared in the data to be compared, and query the data matching table based on the fields to be compared.

[0107] The data matching table is used to determine the domain data type corresponding to each field in the data to be compared.

[0108] Step 320: When it is determined according to the query result to be compared that the data matching table contains a target record that matches the field to be compared, a target field type corresponding to the field to be compared is determined based on the target record.

[0109] The data to be compared contains multiple fields to be compared, and the data matching table needs to be queried based on each field to be compared. If the table identifier (field identifier to be compared) for the data to be compared is T2, the data matching table can be queried based on the field identifier and field to be compared. If a record containing T2 that matches the field to be compared is found, the field data type in that record can be used as the target field type for the field to be compared. If a record containing T2 that matches the field to be compared is not found, the data matching table can be queried a second time based on the field to be compared to obtain the target field type for the field to be compared.

[0110] Step 322: When it is determined based on the query result to be compared that the data matching table does not contain a record matching the field to be compared, the data type to be compared corresponding to the field to be compared is determined based on the data to be compared, and the target field type corresponding to the field to be compared is determined based on the data type to be compared.

[0111] If the target field type corresponding to the to-be-matched field is not obtained after a second query of the data matching table based on the to-be-matched field, the target field type corresponding to the to-be-matched field can be determined based on the data type obtained from the target database. Subsequently, the data conversion table can be queried based on the priority recorded in the data conversion table to determine a serialization method that can serialize the to-be-matched field information corresponding to the to-be-matched field.

[0112] Step 324: When the target field type is a target business type, a target conversion rule is determined in the data conversion table based on the target business type.

[0113] When the target field type is the target business type determined by querying the data matching table, the data conversion table can be directly queried based on the target business type to determine the target conversion rule that can serialize the to-be-matched field information corresponding to the to-be-matched field.

[0114] Step 326: When the target field type is the target data type, search the data conversion table for target conversion rules corresponding to the target business type in order of priority.

[0115] When the target field type is the data type obtained from the source database (target data type), the data conversion table can be queried in combination with the priority recorded in the data conversion table to determine a serialization method that can serialize the to-be-compared field information corresponding to the to-be-compared field, and use the serialization method as the target conversion rule.

[0116] Step 328: Determine the to-be-compared field information corresponding to the to-be-compared field, and convert the to-be-compared field information according to the target conversion rule to obtain the target field information.

[0117] The data in the data to be compared may be stored in the form of key-value pairs, and the value of the field to be compared may be serialized as the field information to be compared to obtain the target field information.

[0118] Step 330: construct a field value data pair based on at least one reference field value included in the reference field information and at least one target field value included in the target field information, and perform consistency comparison on the field value data pair to obtain comparison information.

[0119] Step 332: When it is determined based on the comparison information that the original data and the data to be compared do not pass the data comparison, a target field information pair is determined in the reference field information and the target field information, and the target field information pair is marked with rules in the metadata dimension.

[0120] To summarize, for both the original data and the data to be compared, first determine the serialization method based on the domain data type with stronger business implications. If the domain data type cannot be determined, serialize the original data and the data to be compared based on a more general data type. Then, compare the reference field information obtained from processing the original data and the data to be compared with the target field information, thereby improving the accuracy and observability of the data comparison. This allows for more efficient resolution of data comparison issues across heterogeneous databases.

[0121] Corresponding to the above method embodiment, this specification also provides a data comparison device embodiment, Figure 4 FIG1 shows a schematic diagram of the structure of a data comparison device provided by an embodiment of this specification. Figure 4 As shown, the device includes: A first determination module 402 is configured to determine a data matching table and a data conversion table, and to determine heterogeneous original data and data to be compared; A second determining module 404 is configured to determine, in the data matching table, an initial field type corresponding to an original field in the original data, and to determine, in the data matching table, a target field type corresponding to a to-be-compared field in the to-be-compared data; a conversion module 406 configured to convert the original field information of the original field into reference field information according to an initial conversion rule corresponding to the initial field type in the data conversion table, and to convert the to-be-compared field information of the to-be-compared field into target field information according to a target conversion rule corresponding to the target field type in the data conversion table; The comparison module 408 is configured to perform data comparison on the reference field information and the target field information to obtain comparison information.

[0122] In an optional embodiment, the first determining module 402 is further configured to: Determine at least one type of domain data, and determine data conversion rules and priorities corresponding to the at least one type of domain data; The data conversion table is constructed based on the data conversion rules and priorities corresponding to the at least one type of domain data.

[0123] In an optional embodiment, the second determining module 404 is further configured to: Determining the original field in the original data, and querying the data matching table based on the original field; In a case where it is determined according to the original query result that the data matching table contains an initial record matching the original field, determining the initial field type corresponding to the original field based on the initial record; When it is determined according to the original query result that the data matching table does not contain a record matching the original field, the original data type corresponding to the original field is determined according to the original data, and the initial field type corresponding to the original field is determined according to the original data type.

[0124] In an optional embodiment, the second determining module 404 is further configured to: Determine an original field identifier corresponding to the original field, and query the data matching table based on the original field identifier and the original field; In a case where it is determined based on the original identification query result that the data matching table contains the initial record matching the original field, using the initial record as the original query result; In a case where it is determined based on the original identification query result that the data matching table does not contain a record matching the original field, the data matching table is queried based on the original field.

[0125] In an optional embodiment, the second determining module 404 is further configured to: Determining the field to be compared in the data to be compared, and querying the data matching table based on the field to be compared; In a case where it is determined according to the query result to be compared that the data matching table contains a target record that matches the field to be compared, determining the target field type corresponding to the field to be compared based on the target record; When it is determined according to the query result to be compared that the data matching table does not contain a record matching the field to be compared, the data type to be compared corresponding to the field to be compared is determined according to the data to be compared, and the target field type corresponding to the field to be compared is determined according to the data type to be compared.

[0126] In an optional embodiment, the second determining module 404 is further configured to: Determine a to-be-compared field identifier corresponding to the to-be-compared field, and query the data matching table based on the to-be-compared field identifier and the to-be-compared field; In a case where it is determined based on the query result of the identifier to be compared that the data matching table contains the target record that matches the field to be compared, the target record is used as the query result to be compared; When it is determined based on the query result of the identifier to be compared that the data matching table does not contain a record matching the field to be compared, the data matching table is queried based on the field to be compared.

[0127] In an optional embodiment, the conversion module 406 is further configured to: In a case where the initial field type is an initial business type, determining the initial conversion rule in the data conversion table based on the initial business type; In a case where the initial field type is an initial data type, searching the data conversion table for the initial conversion rule corresponding to the initial business type in order of priority; The original field information corresponding to the original field is determined, and the original field information is converted according to the initial conversion rule to obtain the reference field information.

[0128] In an optional embodiment, the conversion module 406 is further configured to: In a case where the target field type is a target business type, determining the target conversion rule in the data conversion table based on the target business type; In a case where the target field type is a target data type, searching the data conversion table for the target conversion rule corresponding to the target business type in order of priority; The to-be-compared field information corresponding to the to-be-compared field is determined, and the to-be-compared field information is converted according to the target conversion rule to obtain the target field information.

[0129] In an optional embodiment, the conversion module 406 is further configured to: When it is determined that the initial field type includes at least two types of initial field subtypes, updating the rule priority of the data conversion table according to the initial field conversion information corresponding to the at least two types of initial field subtypes; In the case where it is determined that the target field type includes at least two types of target field subtypes, the data conversion table is updated with a rule priority according to target field conversion information corresponding to the at least two types of target field subtypes.

[0130] In an optional embodiment, the comparison module 408 is further configured to: determining at least one reference field value included in the reference field information, and determining at least one target field value included in the target field information; A field value data pair is constructed based on the at least one reference field value and the at least one target field value, and a consistency comparison is performed on the field value data pair to obtain the comparison information.

[0131] In an optional embodiment, the comparison module 408 is further configured to: In a case where it is determined according to the comparison information that the original data and the data to be compared do not pass the data comparison, determining a target field information pair in the reference field information and the target field information; The target field information is marked according to rules in the metadata dimension.

[0132] The data comparison device provided by one embodiment of the present specification determines a data matching table and a data conversion table, and determines heterogeneous original data and data to be compared. In the data matching table, the initial field type corresponding to the original field in the original data is determined, and in the data matching table, the target field type corresponding to the field to be compared in the data to be compared is determined. The data matching table is queried for the original field and the field to be compared respectively to determine the initial field type and the target field type, thereby determining the data type. According to the initial conversion rule corresponding to the initial field type in the data conversion table, the original field information of the original field is converted into reference field information, and according to the target conversion rule corresponding to the target field type in the data conversion table, the field information to be compared of the field to be compared is converted into target field information. By converting the original field information and the field information to be compared respectively, the original field information and the field information to be compared are serialized and processed into reference field information and target field information of the same data format. The reference field information and the target field information are compared in the same data format dimension to obtain comparison information, thereby achieving accurate comparison of the original data and the data to be compared, thereby improving data comparison efficiency.

[0133] The above is a schematic diagram of a data comparison device of this embodiment. It should be noted that the technical solution of the data comparison device and the technical solution of the above-mentioned data comparison method are of the same concept. For details not described in detail in the technical solution of the data comparison device, please refer to the description of the technical solution of the above-mentioned data comparison method.

[0134] Figure 5 The block diagram of a computing device 500 according to one embodiment of the present disclosure is shown. Components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and a database 550 is used to store data.

[0135] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, or a near field communication (NFC) interface.

[0136] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 5 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0137] Computing device 500 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 500 can also be a mobile or stationary server.

[0138] The processor 520 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned data comparison method.

[0139] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned data comparison method are of the same concept. For details not described in detail in the technical scheme of the computing device, please refer to the description of the technical scheme of the above-mentioned data comparison method.

[0140] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which implement the steps of the above-mentioned data comparison method when executed by a processor.

[0141] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned data comparison method are of the same concept. For details not described in detail in the technical scheme of the storage medium, please refer to the description of the technical scheme of the above-mentioned data comparison method.

[0142] An embodiment of the present specification further provides a computer program product, including a computer program or instructions, which implements the steps of the above-mentioned data comparison method when executed by a processor.

[0143] The above is a schematic diagram of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data comparison method described above are based on the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data comparison method described above.

[0144] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0145] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium may be appropriately increased or decreased based on the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0146] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0147] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0148] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of the embodiments described herein. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification.

Claims

1. A data comparison method, characterized in that: include: Determine the data matching table and data conversion table, as well as the heterogeneous original data and data to be compared; Determining, in the data matching table, an initial field type corresponding to an original field in the original data, and determining, in the data matching table, a target field type corresponding to a to-be-compared field in the to-be-compared data; converting the original field information of the original field into reference field information according to the initial conversion rule corresponding to the initial field type in the data conversion table, and converting the to-be-compared field information of the to-be-compared field into target field information according to the target conversion rule corresponding to the target field type in the data conversion table; Perform data comparison on the reference field information and the target field information to obtain comparison information.

2. The data comparison method according to claim 1, characterized in that: The generation of the data conversion table includes: Determine at least one type of domain data, and determine data conversion rules and priorities corresponding to the at least one type of domain data; The data conversion table is constructed based on the data conversion rules and priorities corresponding to the at least one type of domain data.

3. The data comparison method according to claim 1, characterized in that: Determining the initial field type corresponding to the original field in the original data in the data matching table includes: Determining the original field in the original data, and querying the data matching table based on the original field; In a case where it is determined according to the original query result that the data matching table contains an initial record matching the original field, determining the initial field type corresponding to the original field based on the initial record; When it is determined according to the original query result that the data matching table does not contain a record matching the original field, the original data type corresponding to the original field is determined according to the original data, and the initial field type corresponding to the original field is determined according to the original data type.

4. The data comparison method according to claim 3, characterized in that: The querying the data matching table based on the original field further includes: Determine an original field identifier corresponding to the original field, and query the data matching table based on the original field identifier and the original field; In a case where it is determined based on the original identification query result that the data matching table contains the initial record matching the original field, using the initial record as the original query result; In a case where it is determined based on the original identification query result that the data matching table does not contain a record matching the original field, the data matching table is queried based on the original field.

5. The data comparison method according to claim 1, characterized in that: Determining, in the data matching table, a target field type corresponding to the to-be-compared field in the to-be-compared data includes: Determining the field to be compared in the data to be compared, and querying the data matching table based on the field to be compared; In a case where it is determined according to the query result to be compared that the data matching table contains a target record that matches the field to be compared, determining the target field type corresponding to the field to be compared based on the target record; When it is determined according to the query result to be compared that the data matching table does not contain a record matching the field to be compared, the data type to be compared corresponding to the field to be compared is determined according to the data to be compared, and the target field type corresponding to the field to be compared is determined according to the data type to be compared.

6. The data comparison method according to claim 5, characterized in that: The querying of the data matching table based on the field to be compared further includes: Determine a to-be-compared field identifier corresponding to the to-be-compared field, and query the data matching table based on the to-be-compared field identifier and the to-be-compared field; In a case where it is determined based on the query result of the identifier to be compared that the data matching table contains the target record that matches the field to be compared, the target record is used as the query result to be compared; When it is determined based on the query result of the identifier to be compared that the data matching table does not contain a record matching the field to be compared, the data matching table is queried based on the field to be compared.

7. The data comparison method according to claim 1, characterized in that: The converting the original field information of the original field into reference field information according to the initial conversion rule corresponding to the initial field type in the data conversion table includes: In a case where the initial field type is an initial business type, determining the initial conversion rule in the data conversion table based on the initial business type; In a case where the initial field type is an initial data type, searching the data conversion table for the initial conversion rule corresponding to the initial business type in order of priority; The original field information corresponding to the original field is determined, and the original field information is converted according to the initial conversion rule to obtain the reference field information.

8. The data comparison method according to claim 1, characterized in that: The converting the to-be-compared field information of the to-be-compared field into target field information according to the target conversion rule corresponding to the target field type in the data conversion table includes: In a case where the target field type is a target business type, determining the target conversion rule in the data conversion table based on the target business type; In a case where the target field type is a target data type, searching the data conversion table for the target conversion rule corresponding to the target business type in order of priority; The to-be-compared field information corresponding to the to-be-compared field is determined, and the to-be-compared field information is converted according to the target conversion rule to obtain the target field information.

9. The data comparison method according to claim 1, characterized in that: After determining in the data matching table the initial field type corresponding to the original field in the original data, and determining in the data matching table the target field type corresponding to the to-be-compared field in the to-be-compared data, the method further includes: In a case where it is determined that the initial field type includes at least two types of initial field subtypes, updating the rule priority of the data conversion table according to the initial field conversion information corresponding to the at least two types of initial field subtypes; In the case where it is determined that the target field type includes at least two types of target field subtypes, the data conversion table is updated with a rule priority according to target field conversion information corresponding to the at least two types of target field subtypes.

10. The data comparison method according to claim 1, characterized in that: The performing data comparison on the reference field information and the target field information to obtain comparison information includes: determining at least one reference field value included in the reference field information, and determining at least one target field value included in the target field information; A field value data pair is constructed based on the at least one reference field value and the at least one target field value, and a consistency comparison is performed on the field value data pair to obtain the comparison information.

11. The data comparison method according to claim 1, characterized in that: After performing data comparison on the reference field information and the target field information to obtain comparison information, the method further includes: In a case where it is determined according to the comparison information that the original data and the data to be compared do not pass the data comparison, determining a target field information pair in the reference field information and the target field information; The target field information is marked according to rules in the metadata dimension.

12. A computing device, characterized in that include: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the data comparison method described in any one of claims 1 to 11 are implemented.

13. A computer-readable storage medium, characterized in that It stores computer-executable instructions, which, when executed by a processor, implement the steps of the data comparison method described in any one of claims 1 to 11.

14. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the data comparison method according to any one of claims 1 to 11.

Images