Data comparison method and device, computer equipment, readable storage medium and program product
By extracting field information from the data dictionaries of the source and target databases, performing correlation analysis based on business requirements, and determining the specified and ignored fields, the accuracy problem caused by inconsistent table structures in traditional data comparison methods is solved, achieving more accurate data comparison.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-25
- Publication Date
- 2026-04-07
AI Technical Summary
Traditional data comparison methods rely on the consistency of database table structures, which leads to a significant reduction in the accuracy of data verification when the table structures are inconsistent.
By extracting field information from the data dictionaries of the source and target databases, performing correlation analysis based on business requirements, determining specified and ignored field information, and searching for datasets based on this field information, the consistency of the datasets is checked.
It enables more accurate and effective data comparison between databases with different table structures, avoiding misjudgments caused by inconsistent table structures and improving the accuracy and reliability of data comparison.
Smart Images

Figure CN121807835A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data technology, and in particular to a data comparison method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology
[0002] With the rapid development of information technology, scenarios involving cross-system data interaction are increasing, such as in e-commerce and logistics, where multiple systems need to collaborate to complete business loops. Against this backdrop, data comparison technology has emerged to ensure the reliable operation of cross-system business by verifying the consistency of data between different business systems.
[0003] Traditional technologies primarily rely on database table structures for data comparison. For example, in payment reconciliation scenarios, if the transaction record table of a bank's system and the order table of a payment platform have the same structure, differences can be quickly identified by directly comparing field values one by one using SQL (Structured Query Language) queries or ETL (Extract, Transform, Load) tools. However, the effectiveness of this approach is highly dependent on the consistency of the table structures of both parties. In practical applications, inconsistencies in table structures can lead to incorrectly identifying data that was originally consistent as inconsistent, significantly reducing the accuracy of data verification. Summary of the Invention
[0004] Therefore, it is necessary to provide a data comparison method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of data verification in response to the above-mentioned technical problems.
[0005] Firstly, this application provides a data comparison method, including:
[0006] In response to a data comparison command for a source database and a target database, at least one first field information is extracted from the first data dictionary corresponding to the source database, and at least one second field information is extracted from the second data dictionary corresponding to the target database, wherein the table structures of the source database and the target database are different.
[0007] Based on preset business requirements, a correlation analysis is performed on each first field information and each second field information. From the extracted first field information, a first specified field information or a first ignored field information is determined. From the extracted second field information, a second specified field information and a second ignored field information are determined. The correlation between the first ignored field information and the second ignored field information and the preset business requirements is less than a preset correlation threshold.
[0008] The first dataset is retrieved from the source database based on the first specified field information or the first ignored field information, and the second dataset is retrieved from the target database based on the second specified field information or the second ignored field information;
[0009] Check whether the data in the first dataset and the second dataset are consistent.
[0010] In one embodiment, a correlation analysis is performed on each first field information and each second field information based on preset business requirements, and first ignored field information is determined from the extracted first field information, including:
[0011] Identify the source database type;
[0012] Based on the preset mapping relationship between database type and database auto-increment field, the first target auto-increment field information that matches the source database type is filtered out from the first field information. The correlation between the database auto-increment field and the preset business requirements is less than the preset correlation threshold.
[0013] The first target auto-incrementing field information is determined as the first ignored field information;
[0014] Based on pre-defined business requirements, a correlation analysis is performed on each first field and each second field. From the extracted second field information, second ignored field information is determined, including:
[0015] Identify the target database type;
[0016] Based on the preset mapping relationship between database type and database auto-increment field, filter out the second target auto-increment field information that matches the target database type from the second field information;
[0017] The second target auto-increment field information is determined as the second ignored field information.
[0018] In one embodiment, after detecting whether the data in the first dataset and the second dataset are consistent, the method further includes:
[0019] If target data with missing comparison fields is detected in the first dataset, obtain the target field information of the target data;
[0020] In response to the event of adding an ignored field for target field information, the target field information is added as the first ignored field information, and the steps of retrieving the first dataset from the source database based on the first specified field information or the first ignored field information, and retrieving the second dataset from the target database based on the second specified field information or the second ignored field information are returned.
[0021] In one embodiment, detecting whether the data in the first dataset and the second dataset are consistent includes:
[0022] Detect whether the first target dataset to be transformed exists in the first dataset, and whether the second target dataset to be transformed exists in the second dataset;
[0023] If the first target dataset exists in the first dataset, convert the text output format of the data in the first target dataset to obtain the updated first dataset;
[0024] If a second target dataset exists in the second dataset, convert the text output format of the data in the second target dataset to obtain the updated second dataset;
[0025] Check whether the data in the updated first dataset and the updated second dataset are consistent.
[0026] In one embodiment, detecting whether a first target dataset to be transformed exists in the first dataset and whether a second target dataset to be transformed exists in the second dataset includes:
[0027] Based on the source database type, determine the first data type to be converted corresponding to the source database, and find the first target dataset that matches the first data type from the first dataset;
[0028] Based on the target database type, determine the second data type to be converted corresponding to the target database, and find the second target dataset that matches the second data type from the second dataset;
[0029] Convert the text output format of the data in the first target dataset to obtain the updated first dataset, including:
[0030] Based on the first data conversion rule corresponding to the first data type, the text output format of the data in the first target dataset is converted to obtain the updated first dataset.
[0031] Convert the text output format of the data in the second target dataset to obtain the updated second dataset, including:
[0032] Based on the second data conversion rule corresponding to the second data type, the text output format of the data in the second target dataset is converted to obtain the updated second dataset.
[0033] In one embodiment, the first data transformation rule includes multiple first data transformation sub-rules, and the second data transformation rule includes multiple second data transformation sub-rules; after detecting whether the data in the updated first dataset and the updated second dataset are consistent, the method further includes:
[0034] If the first data in the updated first dataset is inconsistent with the second data in the updated second dataset, a data verification message carrying the first data and the second data is pushed, wherein the second data is the comparison object of the first data;
[0035] In response to the feedback operation on the data verification message, query whether there is a target first data transformation sub-rule that matches the first data in each first data transformation sub-rule, and query whether there is a target second data transformation sub-rule that matches the second data in each second data transformation sub-rule;
[0036] If no target first data transformation sub-rule matching the first data is found, a new target first data transformation sub-rule is added to the first data transformation rules;
[0037] If no target second data transformation sub-rule matching the second data is found, a new target second data transformation sub-rule will be added to the second data transformation rules.
[0038] In one embodiment, detecting whether the data in the first dataset and the second dataset are consistent includes:
[0039] Perform a hash operation on the first dataset to obtain the first hash value, and perform a hash operation on the second dataset to obtain the second hash value;
[0040] If the first hash value and the second hash value are inconsistent, the content difference command-line tool is invoked to detect the data that differs between the first dataset and the second dataset.
[0041] Secondly, this application also provides a data comparison device, comprising:
[0042] The extraction module is used to extract at least one first field information from the first data dictionary corresponding to the source database and the target database in response to the data comparison command for the source database and the target database, and to extract at least one second field information from the second data dictionary corresponding to the target database, wherein the table structures of the source database and the target database are different.
[0043] The analysis module is used to perform correlation analysis on each first field information and each second field information based on preset business requirements, determine the first specified field information or the first ignored field information from the extracted first field information, and determine the second specified field information and the second ignored field information from the extracted second field information, wherein the correlation between the first ignored field information and the second ignored field information and the preset business requirements is less than the preset correlation threshold.
[0044] The search module is used to search for a first dataset from the source database based on a first specified field or a first ignored field, and to search for a second dataset from the target database based on a second specified field or a second ignored field.
[0045] The detection module is used to detect whether the data in the first dataset and the second dataset are consistent.
[0046] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0047] In response to a data comparison command for a source database and a target database, at least one first field information is extracted from the first data dictionary corresponding to the source database, and at least one second field information is extracted from the second data dictionary corresponding to the target database, wherein the table structures of the source database and the target database are different.
[0048] Based on preset business requirements, a correlation analysis is performed on each first field information and each second field information. From the extracted first field information, a first specified field information or a first ignored field information is determined. From the extracted second field information, a second specified field information and a second ignored field information are determined. The correlation between the first ignored field information and the second ignored field information and the preset business requirements is less than a preset correlation threshold.
[0049] The first dataset is retrieved from the source database based on the first specified field information or the first ignored field information, and the second dataset is retrieved from the target database based on the second specified field information or the second ignored field information;
[0050] Check whether the data in the first dataset and the second dataset are consistent.
[0051] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:
[0052] In response to a data comparison command for a source database and a target database, at least one first field information is extracted from the first data dictionary corresponding to the source database, and at least one second field information is extracted from the second data dictionary corresponding to the target database, wherein the table structures of the source database and the target database are different.
[0053] Based on preset business requirements, a correlation analysis is performed on each first field information and each second field information. From the extracted first field information, a first specified field information or a first ignored field information is determined. From the extracted second field information, a second specified field information and a second ignored field information are determined. The correlation between the first ignored field information and the second ignored field information and the preset business requirements is less than a preset correlation threshold.
[0054] The first dataset is retrieved from the source database based on the first specified field information or the first ignored field information, and the second dataset is retrieved from the target database based on the second specified field information or the second ignored field information;
[0055] Check whether the data in the first dataset and the second dataset are consistent.
[0056] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:
[0057] In response to a data comparison command for a source database and a target database, at least one first field information is extracted from the first data dictionary corresponding to the source database, and at least one second field information is extracted from the second data dictionary corresponding to the target database, wherein the table structures of the source database and the target database are different.
[0058] Based on preset business requirements, a correlation analysis is performed on each first field information and each second field information. From the extracted first field information, a first specified field information or a first ignored field information is determined. From the extracted second field information, a second specified field information and a second ignored field information are determined. The correlation between the first ignored field information and the second ignored field information and the preset business requirements is less than a preset correlation threshold.
[0059] The first dataset is retrieved from the source database based on the first specified field information or the first ignored field information, and the second dataset is retrieved from the target database based on the second specified field information or the second ignored field information;
[0060] Check whether the data in the first dataset and the second dataset are consistent.
[0061] The aforementioned data comparison method, apparatus, computer equipment, computer-readable storage medium, and computer program product, in response to data comparison instructions for a source database and a target database, extract at least one first field information from a first data dictionary corresponding to the source database and at least one second field information from a second data dictionary corresponding to the target database, thereby achieving field information extraction from source and target databases with different table structures. Furthermore, based on preset business requirements, correlation analysis is performed on each first field and each second field information to determine a first specified field or a first ignored field from the extracted first field information, and a second specified field and a second ignored field from the extracted second field information. The first ignored field information and the second ignored field... The relevance of the information to the preset business requirements is less than a preset relevance threshold, enabling flexible extraction of the required comparison fields from source and target databases with different table structures, breaking through the limitations of traditional table structure consistency. Furthermore, by retrieving the first dataset from the source database based on a first specified field or a first ignored field, and by retrieving the second dataset from the target database based on a second specified field or a second ignored field, targeted extraction of business requirement-related data is achieved. This not only fully preserves key data related to business requirements but also effectively eliminates interference from redundant data unrelated to business needs. Moreover, by detecting the consistency between the data in the first and second datasets, more accurate and effective data comparison can be achieved between databases with different table structures. Thus, even if the table structures of two databases differ, the flexible extraction of fields ensures accurate location and precise comparison of business requirement-related data, effectively avoiding misjudgments caused by inconsistent table structures, thereby significantly improving the accuracy and reliability of data comparison. Attached Figure Description
[0062] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0063] Figure 1 This is a diagram illustrating the application environment of the data comparison method in one embodiment of this application;
[0064] Figure 2 This is a flowchart illustrating a data comparison method in one embodiment of this application;
[0065] Figure 3 This is a flowchart illustrating the process of adding an ignored field in one embodiment of this application;
[0066] Figure 4 This is a flowchart illustrating the data format conversion process in one embodiment of this application;
[0067] Figure 5 This is a flowchart illustrating the process of implementing a new data transformation sub-rule in one embodiment of this application;
[0068] Figure 6 This is a flowchart illustrating the data comparison method in another embodiment of this application;
[0069] Figure 7 This is a structural block diagram of a data comparison device in one embodiment of this application;
[0070] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. It should be noted that existing industry solutions such as software, components, and models may be mentioned in the embodiments of this application. These should be considered exemplary and are intended only to illustrate the feasibility of implementing the technical solutions of this application, but do not imply that the applicant has already used or necessarily used such solutions.
[0072] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations. The acquisition, storage, use and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations.
[0073] The terms "first," "second," etc., used in this application may be used to describe various technical elements, but these technical elements are not limited by these terms. These terms are only used to distinguish the first technical element from the second technical element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.
[0074] The data comparison method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, the first server 102 communicates with the second server 104 via a network. Both the first server 102 and the second server 104 can be configured with corresponding data storage systems, which can store the data that the servers need to process. The data storage system can be integrated onto the server, or it can be located on a cloud or other network server. The first server 102 and the second server 104 can be independent physical servers, or they can be a server cluster or distributed system composed of multiple physical servers, or they can be cloud servers providing cloud computing services.
[0075] The first server 102 and the second server 104 are respectively deployed with two different business systems that are related to each other. For example, the first server 102 is deployed with a banking system, and the second server 104 is deployed with a third-party payment system, a warehouse management system or a transportation scheduling platform. Through the interaction between the first server 102 and the second server 104, the business data between the two different business systems can be compared, thereby ensuring data consistency in the business collaboration process and ensuring the correct execution of cross-system business processes.
[0076] In some feasible implementations, the data comparison method provided in this application can also be applied to data comparison between terminals or between a terminal and a server. The specific application can be determined based on actual circumstances, and this embodiment does not impose any limitations on this. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.
[0077] In one exemplary embodiment, such as Figure 2 As shown, a data comparison method is provided, which can be applied to... Figure 1 Taking the first server 102 as an example, the explanation includes the following steps 202 to 208. Wherein:
[0078] Step 202: In response to the data comparison instruction for the source database and the target database, at least one first field information is extracted from the first data dictionary corresponding to the source database, and at least one second field information is extracted from the second data dictionary corresponding to the target database. The table structures of the source database and the target database are different.
[0079] The source database can refer to the database on the first server, which is used to store business data of the business systems associated with the first server.
[0080] The target database can refer to the database on the second server, which is used to store business data of the business systems associated with the second server.
[0081] The source and target databases can be of the same or different types. For example, the source database can be an Oracle database, and the target database can be a MySQL database (My Structured Query Language, a relational database), or both databases can be MySQL databases.
[0082] When the source and target databases are of the same type, the table structures of the data tables containing the data to be compared can be different.
[0083] In some feasible implementations, the source database and the target database can be databases of two different business systems with business relevance. As an example, the first server 102 can be a server of the first business system, on which the source database is deployed; the second server 104 can be a server of the second business system, on which the target database is deployed; the first server 102 can interact with the second server 104 to compare data in the source database and the target database.
[0084] As an example, the first business system can be a bank system, and the second business system can be a third-party payment system. When a user makes a payment through the third-party payment system, the third-party payment system needs to interact with the bank system to realize the flow of funds. In this scenario, the two systems can ensure the consistency between the flow of funds and business records through regular reconciliation.
[0085] As another example, the first business system can be the e-commerce platform's system, and the second business system can be the logistics system. When a user places an order, the e-commerce platform can push the order information and product information to the logistics system to generate a waybill. The logistics system can also transmit logistics information back in real time, and the e-commerce platform can update the order's logistics status synchronously. In this scenario, both systems can use daily scheduled tasks to compare the order's logistics information to ensure that the logistics information is consistent with the platform's records.
[0086] A data dictionary can refer to a collection of metadata that describes the structure of a database, including information such as table names, field names, field types, constraints, and business meanings.
[0087] The first data dictionary can refer to the set of metadata describing the structure of the source database. The second data dictionary can refer to the set of metadata describing the structure of the target database.
[0088] Field information refers to descriptive information about a field, which may include at least one of the following: field name, data type, length, whether it is nullable, and business meaning. Field information corresponds one-to-one with each field; assuming a database contains N fields, the field information corresponding to each of the N fields can be extracted.
[0089] The first field information can refer to descriptive information about fields in the source database. The second field information can refer to descriptive information about fields in the target database. For example, assuming the first system is a banking system and the second system is a third-party payment system, the first field information contained in the source database of the banking system may include bank transaction number, order number, amount, etc., while the second field information contained in the target database of the third-party payment system may include order number, amount, payment method, etc.
[0090] For example, when the first server receives a request from a user or automated process to perform a consistency comparison of data between the source database and the target database, it immediately triggers and starts the subsequent data comparison process. The data comparison instruction may include basic parameters of the data comparison task, such as connection information between the source and target databases, comparison time range, and business requirement information. After receiving the data comparison instruction, the first server can read the first data dictionary corresponding to the source database from its memory, and then extract the first field information from the first data dictionary.
[0091] As an example, the first server can obtain the second data dictionary of the target database. In this case, after receiving the data comparison instruction, the first server can also directly extract the first field information from the second data dictionary.
[0092] As another example, if the first server cannot obtain the second data dictionary of the target database, after receiving the data comparison instruction, the first server can also send a field information retrieval request to the second server. After detecting the field information retrieval request, the second server can read the second data dictionary corresponding to the target database, extract the second field information from the second data dictionary, and send the extracted second field information to the first server.
[0093] Step 204: Based on the preset business requirements, perform correlation analysis on each first field information and each second field information, determine the first specified field information or the first ignored field information from the extracted first field information, and determine the second specified field information and the second ignored field information from the extracted second field information. The correlation between the first ignored field information and the second ignored field information and the preset business requirements is less than the preset correlation threshold.
[0094] In this context, "business requirements" can refer to the objective of verifying data consistency during cross-system data collaboration to solve specific business problems. This objective determines which data needs to be compared. For example, assuming the business requirement is transaction amount reconciliation, order number data and transaction amount data could be the data that needs to be compared.
[0095] Specified field information can refer to information related to fields highly relevant to the business, including at least one of field names, field identifiers, and field locations. The relevance of the specified field information to the preset business requirements is greater than or equal to a preset relevance threshold. The relevance between each field and the preset business requirements can be determined through at least one of semantic recognition, user-defined methods, or server-defined methods. As an example, in a payment reconciliation scenario, if the preset business requirement is to compare the transaction amount of each order for accuracy, the order number field and the transaction amount field are highly relevant to the preset business requirement and therefore can be used as specified fields.
[0096] The first specified field information can refer to the specified field information corresponding to the source database. The second specified field information can refer to the specified field information corresponding to the target database.
[0097] Ignored field information can refer to information about fields with low business relevance, including at least one of the following: field name, field identifier, field location, etc. The relevance of ignored field information to the preset business requirements is all less than a preset relevance threshold.
[0098] As an example, in the data comparison between the banking system and the third-party payment system, the internal transaction number field in the banking system's data table is only used for internal tracking and has no business meaning. It has low relevance to the preset business requirements, so it can be ignored. The internal transaction number field may not exist in the third-party payment system's data table. Setting it as an ignored field can effectively avoid comparison failures due to missing fields.
[0099] As another example, in data comparison scenarios between Oracle and MySQL databases, MySQL databases typically require primary keys to create tables, while Oracle databases allow tables without primary keys. This can lead to situations where an auto-incrementing primary key field automatically generated in a MySQL table does not have a corresponding field in the Oracle database. Since auto-incrementing primary key fields do not have business meaning and have low relevance to pre-defined business requirements, they can be ignored. By setting this auto-incrementing primary key field as an ignored field, comparison failures caused by differences in database characteristics can be eliminated.
[0100] For example, after the first server extracts the first field information and the second field information, it can compare the correlation between each first field information and each second field information and the preset business requirements. Through correlation analysis, the first field information whose correlation with the preset business requirements is greater than or equal to the preset correlation threshold can be determined as the first designated field information, the first field information whose correlation with the preset business requirements is less than the preset correlation threshold can be determined as the first ignored field information, the second field information whose correlation with the preset business requirements is greater than or equal to the preset correlation threshold can be determined as the second designated field information, and the second field information whose correlation with the preset business requirements is less than the preset correlation threshold can be determined as the second ignored field information.
[0101] As an example, the relevance between field information and preset business requirements can be configured and determined by the user or the first server before data comparison. Before data comparison, the user or the first server can trigger at least one configuration event, either a specified field configuration or an ignored field configuration, for at least one database in the source and target databases, based on preset business requirements. After the first server detects this configuration event, it can obtain at least one of the specified field information of the specified field selected by the user or the first server, and the ignored field information of the ignored field selected by the user or the first server. The specified field information includes first specified field information and second specified field information, and the ignored field information includes first ignored field information and second ignored field information. Furthermore, the first server can set the relevance between the ignored field selected by the user or the first server and the preset business requirements to be less than a preset relevance threshold, and set the relevance between the specified field selected by the user or the first server and the preset business requirements to be greater than or equal to the preset relevance threshold.
[0102] Among them, at least one of the first specified field information, the first ignored field information, the second specified field information, and the second ignored field information is determined according to the configuration event.
[0103] As another example, the relevance between field information and preset business requirements can be determined through semantic recognition and similarity matching. Specifically, during the data comparison process, semantic recognition can be performed on the extracted field information and preset business requirements respectively, and then the similarity between the recognition results can be compared. The similarity is determined as the relevance between the field information and the preset business requirements; the higher the similarity, the higher the relevance between the two.
[0104] In some feasible implementations, if the first server cannot determine the first specified field information and the first ignored field information, it can configure all fields of the source database as specified fields to generate the first specified field information composed of the field information of all fields of the source database.
[0105] In some feasible implementations, if the first server cannot determine the second specified field information and the second ignored field information, it can configure all fields of the target database as specified fields to generate the second specified field information composed of the field information of all fields of the target database.
[0106] Step 206: Find the first dataset from the source database based on the first specified field information or the first ignored field information, and find the second dataset from the target database based on the second specified field information or the second ignored field information.
[0107] The first dataset can refer to the set of data in the source database that needs to be compared. The second dataset can refer to the set of data in the target database that needs to be compared.
[0108] For example, after obtaining the first specified field information or the first ignored field information, the field information in the source database can be filtered according to the obtained first specified field information or the first ignored field information to determine the first field to be compared that matches the field configuration. Then, the first dataset corresponding to the first field to be compared can be exported from the source database according to the derivative command corresponding to the source database. After obtaining the second specified field information or the second ignored field information, the field information in the target database can be filtered according to the obtained second specified field information or the second ignored field information to determine the second field to be compared that matches the field configuration. Then, the second dataset corresponding to the second field to be compared can be exported from the target database according to the derivative command corresponding to the target database.
[0109] If the first business system and the second business system collaborate to complete the target business, as an example, the query method for the second field to be compared may include: the second server first filters data tables related to the target business from the target database and sends the database to the first server; after receiving the data table, the first server filters the field information in the data table according to the obtained second specified field information or second ignored field information to determine the second field to be compared that matches the field configuration.
[0110] As another example, the query method for the second field to be compared may also include: the first server sends the obtained second specified field information or second ignored field information to the second server, and instructs the second server to filter the field information in the target database according to the second specified field information or second ignored field information, and determine the second field to be compared that matches the field configuration.
[0111] If the first business system and the second business system collaborate to complete the target business, as an example, the method of exporting the second dataset corresponding to the second comparison field from the target database may include: the second server first filters the data tables related to the target business from the target database and sends the database to the first server; after receiving the data tables, the first server exports the second dataset corresponding to the second comparison field from the data tables.
[0112] As another example, the method of exporting the second dataset corresponding to the second comparison field from the target database may also include: the first server sends the field information of the determined second comparison field to the second server and instructs the second server to export the second dataset corresponding to the second comparison field from the target database, and the second server sends the exported second dataset to the first server after exporting the second dataset.
[0113] The data export commands can be pre-configured based on the database type and business needs. For example, for Oracle databases, exp, expdp, or custom encapsulation and unloading commands can be used; for MySQL databases, MySQLdump or custom encapsulation and unloading commands can be used. Custom encapsulation and unloading commands refer to customized scripts or command-line tools that users develop based on actual business needs and technology stacks, further modifying the native database export tools.
[0114] Step 208: Check whether the data in the first dataset and the second dataset are consistent.
[0115] For example, after exporting the first dataset and the second dataset, the first server can use preset data consistency comparison rules to compare whether the data in the first dataset and the second dataset are consistent.
[0116] As an example, comparing whether the data in the first dataset and the second dataset are consistent can include: comparing each pair of data in the first dataset and the second dataset one by one according to the mapping relationship between field values; if each pair of data is the same, then the data in the first dataset and the second dataset are considered consistent; if at least one pair of data is different, or at least one pair of data cannot be paired, then the data in the first dataset and the second dataset are considered inconsistent. In the case of detected data inconsistency, the inconsistent data pairs and the data that cannot be paired can be used as the data comparison results, and these results can also be displayed to the user.
[0117] As another example, comparing whether the data in the first dataset and the second dataset are consistent can include: first, hashing the data in the first dataset according to a preset hash algorithm to calculate the first hash value corresponding to the first dataset; then hashing the data in the second dataset according to the same preset hash algorithm to calculate the second hash value corresponding to the second dataset; then comparing the values of the first hash value and the second hash value; if the first hash value and the second hash value are the same, it is determined that the data in the first dataset and the second dataset are consistent; if the first hash value and the second hash value are different, it is determined that the data in the first dataset and the second dataset are inconsistent. If data inconsistency is detected, the data causing the inconsistency can be further identified through a step-by-step comparison as the comparison result, and this comparison result can be displayed to the user. For example, you can first calculate the MD5 (Message-Digest Algorithm 5) value of the first dataset and the MD5 value of the second dataset. If the MD5 values are the same, the data is considered to be consistent. If the MD5 values are inconsistent, the data is considered to be inconsistent. You can then use a diff tool (a software tool used to compare the differences between the contents of two files or directories) to further compare the text and output the inconsistent data as the data comparison result.
[0118] In the aforementioned data comparison method, in response to a data comparison command targeting a source database and a target database, at least one first field is extracted from the first data dictionary corresponding to the source database, and at least one second field is extracted from the second data dictionary corresponding to the target database. This achieves the extraction of field information from source and target databases with different table structures. Furthermore, based on preset business requirements, correlation analysis is performed on each first and second field. From the extracted first field information, a first specified field or a first ignored field is determined; from the extracted second field information, a second specified field and a second ignored field are determined. The first and second ignored field information are correlated with the preset business requirements. With all relevance values below a preset relevance threshold, this system enables flexible extraction of comparison fields required for source and target databases with different table structures, overcoming the limitations of traditional table structure consistency. Furthermore, by retrieving the first dataset from the source database based on a first specified field or a first ignored field, and by retrieving the second dataset from the target database based on a second specified field or a second ignored field, targeted extraction of business-requirement-related data is achieved. This not only fully preserves key data relevant to business needs but also effectively eliminates interference from redundant data unrelated to business needs. Moreover, by detecting the consistency between the first and second datasets, more accurate and effective data comparison can be achieved for databases with different table structures. Thus, even if the table structures of two databases differ, the flexible extraction of fields ensures accurate location and precise comparison of business-requirement-related data, effectively avoiding misjudgments caused by inconsistent table structures, thereby significantly improving the accuracy and reliability of data comparison.
[0119] In one exemplary embodiment, such as Figure 3 As shown, based on preset business requirements, correlation analysis is performed on each first field and each second field, and the first ignored field information is determined from the extracted first field information, including steps 302 to 306. Wherein:
[0120] Step 302: Identify the source database type of the source database.
[0121] It's important to note that while manually configuring specified and ignored fields allows for precise matching of business needs and ensures that the fields included in the comparison align with actual business requirements, the conventional approach is problematic when comparing a large number of fields. The standard procedure often defaults to selecting all fields as the initial specified fields, followed by manual filtering to identify those to be ignored. This approach is highly susceptible to human error, leading to incomplete field configurations, especially with large tables containing dozens of fields. Operators may overlook certain technical or redundant fields that should be ignored. Such incomplete configurations can result in non-critical fields being included in the comparison, increasing unnecessary computational overhead and, more seriously, potentially introducing numerous invalid and inconsistent comparison results, ultimately reducing the accuracy and reliability of the data comparison results.
[0122] The database type can include MySQL, Oracle, etc.
[0123] Different database systems, due to differences in their underlying implementation mechanisms, often contain technical or redundant fields unrelated to the business scenario. Examples include the "ROWID" field in Oracle databases and auto-incrementing primary key fields in MySQL databases. In Oracle, the "ROWID" field is a physical address identifier automatically assigned to each row of data by Oracle. It's a pseudo-column within the database, used only for physical addressing by the storage engine, and doesn't reflect the actual business data content. Even if the business data in two Oracle databases is completely identical, their "ROWID" field values will inevitably differ. In MySQL, auto-incrementing primary key fields are automatically generated and maintained by the database system. When comparing MySQL with other types of databases, because other databases may use completely different primary key generation mechanisms or support tables without primary keys, numerous technical differences will inevitably arise, and these differences are unrelated to business data consistency. Therefore, including these technical or redundant fields unrelated to the business scenario in the data comparison process not only wastes system resources unnecessarily but also leads to a large number of invalid and inconsistent comparison results, severely affecting the accuracy of the comparison results.
[0124] For example, the first server can first obtain the database type information of the source database through the database connection interface, or obtain the database type information of the source database by parsing the connection string, and then determine the source database type of the source database based on the database type information of the source database.
[0125] Step 304: Based on the preset mapping relationship between database type and database auto-increment field, filter out the first target auto-increment field information that matches the source database type from the first field information. The correlation between the database auto-increment field and the preset business requirement is less than the preset correlation threshold.
[0126] In this context, an auto-incrementing field refers to a field that automatically generates a value based on its internal management mechanism when a new record is inserted into a data table. For example, MySQL databases typically require a primary key to create a table, and MySQL database tables automatically generate auto-incrementing primary key fields. These auto-incrementing fields are unrelated to business requirements and are only used for the technical implementation of the database itself. Therefore, the correlation between these auto-incrementing fields and the preset business requirements can be pre-set to be less than a preset correlation threshold.
[0127] For example, during the data comparison process, the first server can determine the first target auto-increment field that matches the source database type based on the preset mapping relationship between the database type and the database auto-increment field, and then filter out the first target auto-increment field information corresponding to the first target auto-increment field from the first field information.
[0128] Step 306: Determine the first target auto-increment field information as the first ignored field information.
[0129] For example, since the correlation between the auto-increment field and the preset business requirement is preset to be less than the preset correlation threshold, during the data comparison process, after the first target auto-increment field information is matched, the first target auto-increment field information will be automatically determined as the first ignored field information because the correlation between it and the preset business requirement will be less than the preset correlation threshold.
[0130] Based on pre-defined business requirements, a correlation analysis is performed on each first field and each second field. From the extracted second field information, second ignored field information is determined, including:
[0131] Identify the target database type; based on the preset mapping relationship between database type and database auto-increment field, filter out the second target auto-increment field information that matches the target database type from the second field information; determine the second target auto-increment field information as the second ignore field information.
[0132] For example, the first server can first obtain the target database type information from the second server. The second server can obtain the target database type information through the database connection interface or by parsing the connection string. Then, the first server can determine the second target auto-incrementing field that matches the target database type based on the preset mapping relationship between the database type and the database auto-incrementing field. Then, the first server can filter out the second target auto-incrementing field information corresponding to the second target auto-incrementing field from the second field information. Since the correlation between the auto-incrementing field and the preset business requirement is preset to be less than the preset correlation threshold, during the data comparison process, after the second target auto-incrementing field information is matched, the second target auto-incrementing field information will be automatically determined as the second ignored field information because the correlation between it and the preset business requirement is less than the preset correlation threshold.
[0133] In this embodiment, by pre-constructing a mapping table based on technical or redundant fields that are irrelevant to the business scenario in each type of database, and automatically matching the corresponding ignored field information based on the database type information during data comparison, non-business-related fields unique to that type of database can be accurately identified and filtered out. This effectively avoids a large number of invalid and inconsistent comparison results caused by the absence or inconsistency of such fields in different business systems, thereby significantly improving the accuracy and reliability of the data comparison results.
[0134] In an exemplary embodiment, after detecting whether the data in the first dataset and the second dataset are consistent, the method further includes:
[0135] If the first dataset contains target data with missing comparison fields, the target field information of the target data is obtained; in response to the event of adding an ignored field for the target field information, the target field information is added as the first ignored field information, and the process of searching for the first dataset from the source database based on the first specified field information or the first ignored field information, and searching for the second dataset from the target database based on the second specified field information or the second ignored field information, is returned until no target data with missing comparison fields is detected, or no event of adding an ignored field is detected.
[0136] It should be noted that while automatically ignoring fields based on a preset mapping table can effectively identify known technical and redundant fields, as database technology continues to evolve, different database systems may introduce new or adjust existing technical and redundant fields. Consequently, these new fields may fail to be identified and incorrectly included in the comparison, resulting in a large number of invalid and inconsistent comparison results, thus reducing the accuracy and reliability of the data comparison results.
[0137] The target data can refer to data that is identified as lacking a corresponding field in the other party's database during the comparison process, thus making data comparison impossible.
[0138] Target field information can refer to the descriptive information of the field to which the target data belongs, that is, the descriptive information of the field that is identified as missing in the other party's database during the comparison process.
[0139] The "Add Ignore Field" event can refer to an operation or signal automatically triggered by a user or the primary server to add a certain field to the mapping table corresponding to the ignored fields.
[0140] For example, during the data comparison process, the first server needs to align the fields of the first and second datasets. When the first server finds that a record in the first dataset contains first data under a certain field, but the corresponding record in the second dataset cannot find that field, it can determine that the first data is missing a comparison field. At this time, the first server can automatically extract the field information of the first data as the target field information. Subsequently, the user or the first server can confirm whether to add the target field information as the first ignored field information corresponding to the source database; the user or the first server can confirm the addition of the target field information as the first ignored field information corresponding to the source database by triggering the add ignored field event; after the first server detects the add ignored field event for the target field information, the first server can establish a mapping relationship between the target field information and the source database, and add the mapping relationship to the preset mapping relationship table between database type and ignored fields. Then, it automatically returns to the steps of finding the first dataset from the source database based on the first specified field information or the first ignored field information, and finding the second dataset from the target database based on the second specified field information or the second ignored field information, and re-performs the data comparison until no target data missing a comparison field is detected, or no add ignored field event is detected.
[0141] As an example, after obtaining the target field information, the target field information can be displayed through the front-end display interface, prompting the user to add the target field information as an ignored field. The user can confirm adding the target field information as an ignored field information through the relevant controls on the previously displayed interface, thereby triggering the Add Ignore Field event.
[0142] As another example, after obtaining the target field information, the first server can automatically identify whether the field corresponding to the target field information was generated by the source database by querying the data source or semantic recognition. If so, it will send a signal to add the target field information as an ignored field information, triggering the addition of the ignored field. If not, the target field information will not be processed.
[0143] Understandably, the same approach can be used to update the second ignored field information corresponding to the target database. As an example, after detecting whether the data in the first dataset and the second dataset are consistent, the method may further include: if second target data lacking a comparison field is detected in the second dataset, obtaining the second target field information of the second target data; in response to a newly added ignored field event for the second target field information, adding the second target field information as the second ignored field information corresponding to the target database, and returning to execute the steps of searching for the first dataset from the source database based on the first specified field information or the first ignored field information, and searching for the second dataset from the target database based on the second specified field information or the second ignored field information, until no second target data lacking a comparison field is detected, or no newly added ignored field event is detected.
[0144] In this embodiment, by verifying the comparison results after data comparison, new technical fields and redundant fields generated by database technology updates are identified, and the mapping relationship between database type and ignored fields is updated. This achieves adaptive processing to the evolution of database technology and can ensure that data comparison maintains high accuracy and reliability in the long term.
[0145] In one exemplary embodiment, such as Figure 4 As shown, the process of detecting whether the data in the first dataset and the second dataset are consistent includes steps 402 to 408. Wherein:
[0146] Step 402: Detect whether the first target dataset to be converted exists in the first dataset, and whether the second target dataset to be converted exists in the second dataset.
[0147] It should be noted that in cross-system data comparison, even if the field names and meanings of two databases are the same, their data formats or storage specifications may differ. For example, the first business system might use the format "YYYY-MM-DD HH:MM:SS" to store time, while the second business system might use a timestamp to store time; the first business system might store monetary data in "yuan" (yuan), while the second business system might store monetary data in "fen" (fen). If the original data is directly compared, even if the business content is the same, inconsistencies in format may lead to misjudgments.
[0148] Before data comparison, format conversion conditions can be set in advance. Format conversion conditions can be used to limit the data range or data type of the data that needs to be converted.
[0149] The first target dataset can refer to the set of data in the first dataset that meets the pre-defined format conversion conditions. The second target dataset can refer to the set of data in the second dataset that meets the pre-defined format conversion conditions.
[0150] For example, during the data comparison process, the first server can detect whether each data in the first dataset and the second dataset meets the preset format conversion conditions, and form the data in the first dataset that meets the preset format conversion conditions into the first target dataset, and form the data in the second dataset that meets the preset format conversion conditions into the second target dataset.
[0151] It is understandable that if the first dataset does not contain data that meets the preset format conversion conditions, the first target dataset can be an empty set. Similarly, if the second dataset does not contain data that meets the preset format conversion conditions, the second target dataset can be an empty set.
[0152] Step 404: If the first target dataset exists in the first dataset, convert the text output format of the data in the first target dataset to obtain the updated first dataset.
[0153] For example, when the existence of the first target dataset is detected in the first dataset, the first server can convert the text output format of the data in the first target dataset according to the preset data conversion rules, obtain the format-converted data, and replace the corresponding unconverted data in the first dataset with the format-converted data to obtain the updated first dataset.
[0154] The preset data conversion rules refer to pre-defined data format conversion logic used to unify the original data in the database into the target format. For example, monetary data can be formatted as "X.XX", which includes the integer part and two decimal places. However, some monetary data may be represented as ".XX" when the integer part is 0, meaning the integer part is not displayed. In this case, it can be converted to the "X.XX" format through format conversion.
[0155] Understandably, when the first dataset includes multiple types of data, the data transformation rules for different types of data may differ. For example, the data transformation rules for time data and monetary data are obviously different.
[0156] Step 406: If the second target dataset exists in the second dataset, convert the text output format of the data in the second target dataset to obtain the updated second dataset.
[0157] For example, if a second target dataset is detected in the second dataset, the first server can convert the text output format of the data in the second target dataset according to a preset data conversion rule to obtain the format-converted data, and replace the corresponding unconverted data in the second dataset with the format-converted data to obtain the updated second dataset.
[0158] Step 408: Check whether the data in the updated first dataset and the updated second dataset are consistent.
[0159] For example, after updating the first dataset and the second dataset, the first server can use a preset data consistency comparison rule to compare whether the data in the updated first dataset and the updated second dataset are consistent.
[0160] In this embodiment, format conversion overcomes the technical deficiency of misjudgment caused by inconsistent data representation formats. This not only ensures accurate identification of substantially consistent data even when there are differences in time format, monetary unit, encoding method, etc., but also significantly improves the accuracy and reliability of comparison results. Furthermore, it enhances the flexibility and maintainability of data comparison; when a new database type or format rule changes, only the mapping configuration needs to be updated, without modifying the core comparison logic, making it suitable for complex and ever-changing cross-system data collaboration scenarios.
[0161] In an exemplary embodiment, detecting whether a first target dataset to be transformed exists in a first dataset, and whether a second target dataset to be transformed exists in a second dataset, includes:
[0162] Based on the source database type, determine the first data type to be converted corresponding to the source database, and find the first target dataset that matches the first data type from the first dataset; based on the target database type, determine the second data type to be converted corresponding to the target database, and find the second target dataset that matches the second data type from the second dataset.
[0163] For example, the first server can first obtain the database type information of the source database through the database connection interface, or obtain the database type information of the source database by parsing the connection string, and determine the source database type of the source database based on the database type information of the source database, thereby finding the first target dataset that matches the first data type from the first dataset; the first server can also first obtain the database type information of the target database from the second server, determine the target database type of the target database based on the database type information of the target database, and thereby find the second target dataset that matches the second data type from the second dataset, wherein the second server can obtain the target database type information of the target database through the database connection interface, or obtain the database type information of the target database by parsing the connection string.
[0164] Convert the text output format of the data in the first target dataset to obtain the updated first dataset, including:
[0165] Based on the first data conversion rule corresponding to the first data type, the text output format of the data in the first target dataset is converted to obtain the updated first dataset.
[0166] Different databases typically have different data formats. Therefore, different data conversion rules can be set for different database types. When comparing data, the corresponding data conversion rules can be quickly matched by database type and data type to achieve rapid conversion and unification of data formats.
[0167] For example, after determining the first data type and the first target dataset, the first server can query the first data conversion rule corresponding to the first data type from the preset mapping relationship between data types and data conversion rules, and then convert the text output format of the data in the first target dataset according to the first data conversion rule to obtain the format-converted data. The format-converted data is then used to replace the corresponding unconverted data in the first dataset to obtain the updated first dataset.
[0168] Convert the text output format of the data in the second target dataset to obtain the updated second dataset, including:
[0169] Based on the second data conversion rule corresponding to the second data type, the text output format of the data in the second target dataset is converted to obtain the updated second dataset.
[0170] For example, after determining the second data type and the second target dataset, the first server can query the second data conversion rule corresponding to the second data type from the preset mapping relationship between data types and data conversion rules, and then convert the text output format of the data in the second target dataset according to the second data conversion rule to obtain the format-converted data. The format-converted data is then used to replace the corresponding unconverted data in the second dataset to obtain the updated second dataset.
[0171] In this embodiment, by quickly matching the database type and data type with the corresponding data conversion rules, the rapid conversion and unification of data formats are achieved.
[0172] In one exemplary embodiment, such as Figure 5 As shown, the first data transformation rule includes multiple first data transformation sub-rules, and the second data transformation rule includes multiple second data transformation sub-rules; after detecting whether the data in the updated first dataset and the updated second dataset are consistent, the method further includes steps 502 to 508. Wherein:
[0173] Step 502: If the first data in the updated first dataset is inconsistent with the second data in the updated second dataset, push a data verification message carrying the first data and the second data, wherein the second data is the comparison object of the first data.
[0174] It should be noted that in cross-system data comparison, even if data transformation rules based on database type are adopted, some fields may still not be aligned after transformation due to the introduction of new fields, changes in business logic, or incomplete coverage of transformation rules. This may lead to misjudgment as data inconsistency, reducing the accuracy and reliability of data comparison results.
[0175] Here, "first data" refers to the data in the updated first dataset that was found to be inconsistent during comparison. "Second data" is the data being compared with the first data. That is, when comparing the updated first dataset with the updated second dataset, inconsistencies were detected between the first and second data.
[0176] For example, after performing a data consistency check on the updated first dataset and the updated second dataset, the first server detects that the first data in the updated first dataset is inconsistent with the corresponding second data in the updated second dataset. In this case, the first server generates a data verification message based on the first and second data and pushes the data verification message to the front-end interface for visualization.
[0177] Step 504: In response to the feedback operation for the data verification message, query whether there is a target first data transformation sub-rule that matches the first data in each first data transformation sub-rule, and query whether there is a target second data transformation sub-rule that matches the second data in each second data transformation sub-rule.
[0178] Feedback operations can refer to user actions such as confirmation, correction, or configuration commands performed in response to inconsistencies in the displayed data verification messages. For example, users can click the "Mark as Needing Conversion" button on the front-end interface or select the "Add Conversion Rule" option.
[0179] For example, when a user provides feedback on inconsistencies in the displayed data verification message, the first server can respond to this feedback by initiating a rule verification and completion process. The first server can first perform a matching query in the corresponding first data transformation sub-rules of the source database based on data information such as the source field, data type, and data characteristics of the first data to determine if a defined target first data transformation sub-rule applicable to that field exists. The first server can also perform a matching query in the corresponding second data transformation sub-rules of the target database based on data information such as the source field, data type, and data characteristics of the second data to determine if a defined target second data transformation sub-rule applicable to that field exists. The query results can be used to determine whether the current discrepancy is due to a missing rule.
[0180] Step 506: If no target first data transformation sub-rule matching the first data is found, a new target first data transformation sub-rule is added to the first data transformation rules.
[0181] For example, if the first server does not find a target first data transformation sub-rule that matches the first data, it indicates that the current transformation rule system lacks processing logic for that field. In this case, the first server can generate a transformation rule applicable to the field based on a preset strategy or guidance provided by the user in the feedback operation, and add the generated transformation rule as the target first data transformation sub-rule corresponding to the first data to the first data transformation rule set of the source database, thus completing the dynamic expansion of the rule base.
[0182] Step 508: If no target second data transformation sub-rule matching the second data is found, a new target second data transformation sub-rule is added to the second data transformation rules.
[0183] For example, if the first server does not find a target second data transformation sub-rule that matches the second data, it indicates that the current transformation rule system lacks processing logic for that field. In this case, the first server can generate a transformation rule applicable to the field based on a preset strategy or guidance provided by the user in the feedback operation, and add the generated transformation rule as the target second data transformation sub-rule corresponding to the second data to the second data transformation rule set of the target database, thus completing the dynamic expansion of the rule base.
[0184] In this embodiment, when an inconsistency caused by missing conversion rules is detected, a rule verification and completion process is triggered through user feedback. The missing data conversion sub-rules are automatically identified and supplemented, which can effectively solve the problem of false alarms caused by incomplete rule coverage, significantly improve the accuracy and scalability of data comparison, and enhance the long-term stability and adaptability of data comparison.
[0185] In some feasible implementations, detecting whether the data in the first dataset and the second dataset are consistent includes:
[0186] Perform a hash operation on the first dataset to obtain the first hash value, and perform a hash operation on the second dataset to obtain the second hash value. If the first hash value and the second hash value are inconsistent, call the content difference command-line tool to detect the data that differs between the first dataset and the second dataset.
[0187] For example, after determining the first dataset and the second dataset, a hash operation can be performed on all data in the first dataset to obtain the first hash value corresponding to the first dataset; a hash operation can also be performed on all data in the second dataset to obtain the second hash value corresponding to the second dataset; then, the numerical values of the first hash value and the second hash value are compared; if they are the same, it indicates that the data in the first dataset and the data in the second dataset are completely consistent, and a data comparison result with no difference can be generated; if they are different, it indicates that the data in the first dataset and the data in the second dataset are not completely consistent, and a content difference command-line tool can be called to detect the data that differs between the first dataset and the second dataset, and the data that differs between the first dataset and the second dataset can be output as the data comparison result.
[0188] In some feasible implementations, the hash value can be an MD5 value; the content difference command-line tool can be a diff tool.
[0189] In this embodiment, a consistency check at the dataset level is first performed on the first dataset and the second dataset using hash operations. Only when differences are determined between the datasets is a comparison performed on each data item. This avoids performing a large number of unnecessary comparisons on each item when the data is completely consistent, thereby significantly reducing computational overhead and improving the efficiency of data comparison.
[0190] In some feasible implementations, such as Figure 6 As shown, before extracting at least one first field information from the first data dictionary corresponding to the source database and at least one second field information from the second data dictionary corresponding to the target database, the process may further include:
[0191] Step 602: Set the database type for the tables to be compared. Currently, the main types are Oracle and MySQL. This configuration mainly involves reading the database's data dictionary. For Oracle, the database dictionary can use `user_tab_columns` to read the column numbers and data types of all data columns in the table. For MySQL, the database dictionary can use `information_schema.columns` to read the column numbers and data types of all data columns in the table. If a new database type requires a different data dictionary to read the table's column information, a custom data dictionary corresponding to that database type can be defined.
[0192] Step 604: Configure the custom command corresponding to the database derivative. The general format of the derivative command can be xxx.sh "SQL statement", where xxx.sh can be defined by the user. For example, Oracle can be executed using exp, expdp, or a user-defined decomposition and decomposition command; MySQL can be executed using MySQLdump or a user-defined decomposition and decomposition command.
[0193] Step 606: Configure the conversion method for data types exported to text in the database. Since the output text results may differ between different databases and data types, it's necessary to configure corresponding conversion methods. For example, with Oracle's NUMBER type, when decimal precision is included and the integer is 0, the current script might export it as .XX text. In this case, you need to convert the Oracle NUMBER data type using `to_char(column_name, 'fm9999999999999990.00')`. Similarly, for DATE or TIMESTAMP types, you can also use `to_char` or similar conversion methods depending on the inconsistent display of the exported script.
[0194] Step 608: Configure the database data retrieval range. This includes the tables in data retrieval queues 1 to N, as well as the comparison columns and ignore columns. It should be noted that each column in a data table typically represents an independent field. This configuration supports configuring multiple database tables for data comparison at once, allowing for serial or parallel data retrieval. Serial comparison avoids high I / O issues caused by retrieval from multiple tables in the same database, while parallel comparison improves efficiency. Users can choose according to their needs. Simultaneously with table configuration, the corresponding comparison columns can be configured. The default is to compare all columns, but users can filter columns as needed. If comparison columns exist, they can be read from `user_tab_columns` or `information_schema.columns`, using `column_id in` or `ordinal_position in` to specify the columns to be compared. If ignore columns exist, they can be ignored using `not in`. Since Oracle can create tables without a primary key, while most domestic databases based on MySQL require a primary key to create a table, the MySQL table data may contain an auto-incrementing primary key. In this case, this column needs to be ignored when comparing the table data.
[0195] Step 610: Configure the data comparison method. The data comparison method can be: export the MD5 values of data files from the same table in different databases; if the MD5 values match, the comparison result is considered consistent, and no further text comparison using a diff tool is needed; if the MD5 values do not match, the comparison result is considered inconsistent, and a text comparison using a diff tool is required to output the inconsistent results to a result file. Users can visually see whether the comparison is consistent and which data is inconsistent.
[0196] Step 612: Configure the output path for the comparison results.
[0197] In some feasible implementations, data comparison methods can be applied to reconciliation within a bank's internal systems. In this scenario, different business systems may use different daily cutoff times: some systems use 23:00 as the daily cutoff time (forming an accounting cycle from 23:00 on day T-1 to 23:00 on day T), while other systems use 0:00 as the daily cutoff time (forming an accounting cycle from 0:00 on day T to 24:00 on day T). Since reconciliation files are typically generated according to the system's accounting cycle, reconciliation between systems with different cycles requires special handling. In this scenario, data comparison methods may include: obtaining reconciliation files daily; selecting appropriate transaction type rules based on system differences to ensure that both payment and return transactions participate in the reconciliation; verifying the transaction number against the original transaction amount to achieve reconciliation; and timeliness: adopting a T+1 daily reconciliation model. This method ensures reconciliation accuracy while also considering processing efficiency, making it suitable for inter-system reconciliation scenarios with high transaction volumes, high accuracy requirements, but relatively relaxed timeliness requirements.
[0198] In some feasible implementations, data comparison methods can be applied to reconciliation between banks and third-party payment institutions. In this scenario, due to differences in business roles, the transaction success times recorded by the two systems may differ. For example, in a UnionPay gateway payment transaction involving both the card issuer and the acquiring side, when the issuing system acts as the card issuer, another bank acts as the acquiring side, and UnionPay acts as the third-party institution, the issuing system uses the time of successful card deduction as the transaction success registration time, while UnionPay uses the time of successful response from the acquiring side. Due to various factors, the transaction time records of the two sides may differ. Since UnionPay supports generating reconciliation transaction details hourly, data comparison methods can include: obtaining reconciliation files hourly; selecting appropriate transaction type rules based on system differences to ensure that both payment and return transactions participate in reconciliation; aligning transaction numbers or order numbers, transaction types, and original transaction amounts; supporting cross-day reconciliation; and timeliness: adopting a T+1 daily reconciliation mode. When enabling cross-day reconciliation, it is necessary to select an appropriate reconciliation time point based on actual business needs to reduce errors. For example, if reconciliation is performed at 8:00 AM on T+1 day, the transaction history of this party on day T can be compared with the transaction history of the reconciling party from 0:00 AM on day T to 8:00 AM on day T+1. Similarly, the transaction history of the reconciling party on day T can be matched with the transaction history of this party over days TN (where N is the number of days the historical transaction history is stored in this party's system). This method can meet the timeliness and accuracy requirements of most system reconciliations and is suitable for a wide range of reconciliation scenarios. However, it is not suitable for inter-system reconciliation with extremely high transaction volumes and a large number of daily errors, and cross-day reconciliation places high demands on system equipment performance.
[0199] In some feasible implementations, data comparison methods can be applied to banks and third-party payment institutions. In this scenario, the transaction times registered by both parties are highly consistent in most business scenarios. After the bank completes the transaction and responds successfully, the third-party payment institution receives the response, simultaneously registers the transaction as successful, and sends feedback to the customer. The third-party payment institution generates a reconciliation file at a set time every morning, containing elements such as order number, transaction type, order status, transaction amount (for returned transactions, the amount requested for a refund), and transaction time. When the third-party payment institution's reconciliation statement is entered into the database, records with unsuccessful order statuses can be marked as "not participating in reconciliation." Data comparison methods can include: obtaining reconciliation files daily; including both payment and return transactions in the reconciliation process; reconciliation based on order number, transaction type, and transaction amount; and timeliness: using a T+1 daily reconciliation model. This method, under the premise that the reconciliation file generation cycle of both parties is consistent and the transaction times are highly synchronized, is suitable for reconciliation tasks with large amounts of data, and can achieve efficient reconciliation processing while ensuring high reconciliation accuracy.
[0200] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.
[0201] Based on the same inventive concept, this application also provides a data comparison apparatus for implementing the data comparison method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data comparison apparatus embodiments provided below can be found in the limitations of the data comparison method described above, and will not be repeated here.
[0202] In one exemplary embodiment, such as Figure 7 As shown, a data comparison device is provided, including: an extraction module 702, an analysis module 704, a search module 706, and a detection module 708, wherein:
[0203] Extraction module 702 is used to extract at least one first field information from the first data dictionary corresponding to the source database and at least one second field information from the second data dictionary corresponding to the target database in response to the data comparison instruction for the source database and the target database. The table structures of the source database and the target database are different.
[0204] Analysis module 704 is used to perform correlation analysis on each first field information and each second field information based on preset business requirements, determine first specified field information or first ignored field information from the extracted first field information, and determine second specified field information and second ignored field information from the extracted second field information, wherein the correlation between the first ignored field information and the second ignored field information and the preset business requirements is less than a preset correlation threshold.
[0205] The lookup module 706 is used to look up a first dataset from the source database based on a first specified field or a first ignored field, and to look up a second dataset from the target database based on a second specified field or a second ignored field.
[0206] The detection module 708 is used to detect whether the data in the first dataset and the second dataset are consistent.
[0207] In one exemplary embodiment, the analysis module includes:
[0208] The first identification unit is used to identify the source database type of the source database;
[0209] The first mapping unit is used to filter out the first target auto-increment field information that matches the source database type from the first field information according to the preset mapping relationship between the database type and the database auto-increment field. The correlation between the database auto-increment field and the preset business requirements is less than the preset correlation threshold.
[0210] The first determining unit is used to determine the first target auto-incrementing field information as the first ignored field information;
[0211] The second identification unit is used to identify the target database type of the target database.
[0212] The second mapping unit is used to filter out the second target auto-increment field information that matches the target database type from the second field information according to the preset mapping relationship between the database type and the database auto-increment field;
[0213] The second determining unit is used to determine the second target auto-increment field information as the second ignored field information.
[0214] In one exemplary embodiment, the apparatus further includes:
[0215] The acquisition module is used to acquire the target field information of the target data after detecting whether the data in the first dataset and the second dataset are consistent, and when the target data in the first dataset is found to be missing a comparison field.
[0216] The first new module is used to respond to the event of adding an ignored field for the target field information, add the target field information as the first ignored field information, and return to the steps of performing the steps of finding the first dataset from the source database based on the first specified field information or the first ignored field information, and finding the second dataset from the target database based on the second specified field information or the second ignored field information.
[0217] In one exemplary embodiment, the detection module includes:
[0218] The first detection unit is used to detect whether there is a first target dataset to be transformed in the first dataset, and whether there is a second target dataset to be transformed in the second dataset;
[0219] The first conversion unit is used to convert the text output format of the data in the first target dataset to obtain the updated first dataset, provided that the first target dataset exists in the first dataset.
[0220] The second conversion unit is used to convert the text output format of the data in the second target dataset to obtain the updated second dataset when the second target dataset exists in the second dataset.
[0221] The second detection unit is used to detect whether the data in the updated first dataset and the updated second dataset are consistent.
[0222] In one exemplary embodiment, the first detection unit is further configured to:
[0223] Based on the source database type, determine the first data type to be converted corresponding to the source database, and find the first target dataset that matches the first data type from the first dataset;
[0224] Based on the target database type, determine the second data type to be converted corresponding to the target database, and find the second target dataset that matches the second data type from the second dataset;
[0225] The first conversion unit is also used for:
[0226] Based on the first data conversion rule corresponding to the first data type, the text output format of the data in the first target dataset is converted to obtain the updated first dataset.
[0227] The second conversion unit is also used for:
[0228] Based on the second data conversion rule corresponding to the second data type, the text output format of the data in the second target dataset is converted to obtain the updated second dataset.
[0229] In an exemplary embodiment, the first data transformation rule includes multiple first data transformation sub-rules, and the second data transformation rule includes multiple second data transformation sub-rules; the apparatus further includes:
[0230] The push module is used to push a data verification message carrying the first data and the second data when the first data in the updated first dataset is inconsistent with the second data in the updated second dataset, wherein the second data is the comparison object of the first data;
[0231] The query module is used to respond to feedback operations on data verification messages, query whether there is a target first data transformation sub-rule that matches the first data in each first data transformation sub-rule, and query whether there is a target second data transformation sub-rule that matches the second data in each second data transformation sub-rule;
[0232] The second newly added module is used to add a target first data transformation sub-rule to the first data transformation rule if no target first data transformation sub-rule matching the first data is found.
[0233] The third newly added module is used to add a target second data transformation sub-rule to the second data transformation rule if no target second data transformation sub-rule matching the second data is found.
[0234] In one exemplary embodiment, the detection module further includes:
[0235] The processing unit is used to perform a hash operation on the first dataset to obtain a first hash value, and to perform a hash operation on the second dataset to obtain a second hash value;
[0236] The third detection unit is used to call the content difference command-line tool to detect data that differs between the first and second datasets when the first hash value and the second hash value are inconsistent.
[0237] Each module in the aforementioned data comparison device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0238] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a data comparison method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.
[0239] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0240] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.
[0241] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.
[0242] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0243] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0244] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0245] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A data comparison method, characterized in that, The method includes: In response to a data comparison instruction for a source database and a target database, at least one first field information is extracted from a first data dictionary corresponding to the source database, and at least one second field information is extracted from a second data dictionary corresponding to the target database, wherein the table structures of the source database and the target database are different; Based on preset business requirements, a correlation analysis is performed on each of the first field information and each of the second field information. From the extracted first field information, a first specified field information or a first ignored field information is determined. From the extracted second field information, a second specified field information and a second ignored field information are determined. The correlation between the first ignored field information and the second ignored field information and the preset business requirements is less than a preset correlation threshold. A first dataset is retrieved from the source database based on the first specified field information or the first ignored field information, and a second dataset is retrieved from the target database based on the second specified field information or the second ignored field information; Check whether the data in the first dataset and the second dataset are consistent.
2. The method according to claim 1, characterized in that, Based on preset business requirements, a correlation analysis is performed on each of the first field information and each of the second field information. From the extracted first field information, first ignored field information is determined, including: Identify the source database type of the source database; Based on the preset mapping relationship between database type and database auto-increment field, the first target auto-increment field information that matches the source database type is filtered out from the first field information, and the correlation between the database auto-increment field and the preset business requirement is less than the preset correlation threshold. The first target auto-incrementing field information is determined as the first ignored field information; Based on preset business requirements, a correlation analysis is performed on each of the first field information and each of the second field information. From the extracted second field information, second ignored field information is determined, including: Identify the target database type of the target database; Based on the preset mapping relationship between database types and database auto-incrementing fields, filter out the second target auto-incrementing field information that matches the target database type from the second field information; The second target auto-increment field information is determined as the second ignored field information.
3. The method according to claim 1, characterized in that, After detecting whether the data in the first dataset and the second dataset are consistent, the method further includes: If the first dataset contains target data that lacks a comparison field, the target field information of the target data is obtained. In response to the newly added ignored field event for the target field information, the target field information is added as the first ignored field information, and the process returns to the steps of finding the first dataset from the source database based on the first specified field information or the first ignored field information, and finding the second dataset from the target database based on the second specified field information or the second ignored field information, until no target data lacking a comparison field is detected, or no newly added ignored field event is detected.
4. The method according to claim 1, characterized in that, The step of detecting whether the data in the first dataset and the second dataset are consistent includes: Detect whether the first target dataset to be transformed exists in the first dataset, and whether the second target dataset to be transformed exists in the second dataset; If the first target dataset exists in the first dataset, convert the text output format of the data in the first target dataset to obtain the updated first dataset; If the second target dataset exists in the second dataset, convert the text output format of the data in the second target dataset to obtain the updated second dataset; Check whether the data in the updated first dataset and the updated second dataset are consistent.
5. The method according to claim 4, characterized in that, The step of detecting whether there is a first target dataset to be transformed in the first dataset and whether there is a second target dataset to be transformed in the second dataset includes: Based on the source database type of the source database, determine the first data type to be converted corresponding to the source database, and find the first target dataset that matches the first data type from the first dataset; Based on the target database type of the target database, determine the second data type to be converted corresponding to the target database, and find the second target dataset that matches the second data type from the second dataset; The step of converting the text output format of the data in the first target dataset to obtain the updated first dataset includes: According to the first data conversion rule corresponding to the first data type, the text output format of the data in the first target dataset is converted to obtain the updated first dataset. The step of converting the text output format of the data in the second target dataset to obtain the updated second dataset includes: According to the second data conversion rule corresponding to the second data type, the text output format of the data in the second target dataset is converted to obtain the updated second dataset.
6. The method according to claim 5, characterized in that, The first data transformation rule includes multiple first data transformation sub-rules, and the second data transformation rule includes multiple second data transformation sub-rules; after detecting whether the data in the updated first dataset and the updated second dataset are consistent, the method further includes: If the first data in the updated first dataset is inconsistent with the second data in the updated second dataset, a data verification message carrying the first data and the second data is pushed, wherein the second data is the comparison object of the first data; In response to the feedback operation for the data verification message, query whether there is a target first data transformation sub-rule that matches the first data in each of the first data transformation sub-rules, and query whether there is a target second data transformation sub-rule that matches the second data in each of the second data transformation sub-rules; If no target first data transformation sub-rule matching the first data is found, then the target first data transformation sub-rule is added to the first data transformation rule; If no target second data transformation sub-rule matching the second data is found, then the target second data transformation sub-rule is added to the second data transformation rules.
7. The method according to any one of claims 1 to 5, characterized in that, The step of detecting whether the data in the first dataset and the second dataset are consistent includes: Perform a hash operation on the first dataset to obtain a first hash value, and perform a hash operation on the second dataset to obtain a second hash value; If the first hash value and the second hash value are inconsistent, the content difference command-line tool is invoked to detect the data that differs between the first dataset and the second dataset.
8. A data comparison device, characterized in that, The device includes: An extraction module is used to extract at least one first field information from a first data dictionary corresponding to the source database and a target database in response to a data comparison instruction for the source database and the target database, and to extract at least one second field information from a second data dictionary corresponding to the target database, wherein the table structures of the source database and the target database are different. The analysis module is used to perform correlation analysis on each of the first field information and each of the second field information based on preset business requirements, determine the first specified field information or the first ignored field information from the extracted first field information, and determine the second specified field information and the second ignored field information from the extracted second field information, wherein the correlation between the first ignored field information and the second ignored field information and the preset business requirements is less than the preset correlation threshold. The search module is used to search for a first dataset from the source database based on the first specified field information or the first ignored field information, and to search for a second dataset from the target database based on the second specified field information or the second ignored field information; The detection module is used to detect whether the data in the first dataset and the second dataset are consistent.
9. The apparatus according to claim 8, characterized in that, The analysis module includes: The first identification unit is used to identify the source database type of the source database; The first mapping unit is used to filter out the first target auto-increment field information that matches the source database type from the first field information according to the preset mapping relationship between the database type and the database auto-increment field, wherein the correlation between the database auto-increment field and the preset business requirement is less than the preset correlation threshold. The first determining unit is used to determine the first target auto-incrementing field information as the first ignored field information; The second identification unit is used to identify the target database type of the target database; The second mapping unit is used to filter out second target auto-incrementing field information that matches the target database type from the second field information according to the preset mapping relationship between the database type and the database auto-incrementing field; The second determining unit is used to determine the second target auto-increment field information as the second ignored field information.
10. The apparatus according to claim 8, characterized in that, The device further includes: The acquisition module is used to acquire target field information of the target data after detecting whether the data in the first dataset and the second dataset are consistent, in the case that the target data in the first dataset is missing a comparison field; The first addition module is used to respond to the addition of an ignored field event for the target field information, add the target field information as the first ignored field information, and return to perform the steps of finding the first dataset from the source database based on the first specified field information or the first ignored field information, and finding the second dataset from the target database based on the second specified field information or the second ignored field information.
11. The apparatus according to claim 8, characterized in that, The detection module includes: The first detection unit is used to detect whether there is a first target dataset to be converted in the first dataset, and whether there is a second target dataset to be converted in the second dataset; The first conversion unit is used to convert the text output format of the data in the first target dataset to obtain the updated first dataset when the first target dataset exists in the first dataset. The second conversion unit is used to convert the text output format of the data in the second target dataset to obtain the updated second dataset when the second target dataset exists in the second dataset. The second detection unit is used to detect whether the data in the updated first dataset and the updated second dataset are consistent.
12. The apparatus according to claim 11, characterized in that, The first detection unit is further configured to: Based on the source database type of the source database, determine the first data type to be converted corresponding to the source database, and find the first target dataset that matches the first data type from the first dataset; Based on the target database type of the target database, determine the second data type to be converted corresponding to the target database, and find the second target dataset that matches the second data type from the second dataset; The first conversion unit is further configured to: According to the first data conversion rule corresponding to the first data type, the text output format of the data in the first target dataset is converted to obtain the updated first dataset. The second conversion unit is further configured to: According to the second data conversion rule corresponding to the second data type, the text output format of the data in the second target dataset is converted to obtain the updated second dataset.
13. The apparatus according to claim 12, characterized in that, The first data transformation rule includes multiple first data transformation sub-rules, and the second data transformation rule includes multiple second data transformation sub-rules; the device further includes: The push module is used to push a data verification message carrying the first data and the second data when the first data in the updated first dataset is inconsistent with the second data in the updated second dataset, wherein the second data is the comparison object of the first data; The query module is used to respond to the feedback operation of the data verification message, query whether there is a target first data transformation sub-rule matching the first data in each of the first data transformation sub-rules, and query whether there is a target second data transformation sub-rule matching the second data in each of the second data transformation sub-rules; The second newly added module is used to add the target first data transformation sub-rule to the first data transformation rule if no target first data transformation sub-rule matching the first data is found. The third newly added module is used to add the target second data transformation sub-rule to the second data transformation rule if no target second data transformation sub-rule matching the second data is found.
14. The apparatus according to any one of claims 8 to 13, characterized in that, The detection module also includes: The processing unit is used to perform a hash operation on the first dataset to obtain a first hash value, and to perform a hash operation on the second dataset to obtain a second hash value; The third detection unit is used to call the content difference command-line tool to detect data that differs between the first dataset and the second dataset when the first hash value and the second hash value are inconsistent.
15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.
16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.
17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.