Database data verification method and device, equipment and medium

Abstract

The invention relates to the technical field of data processing, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a database data verification method which comprises the following steps: configuring source database connection parameters and target database connection parameters, setting comparison task parameters, and associating the comparison task parameters with corresponding database connection parameters; determining a source table and a target table corresponding to the source table identifier and the target table identifier, executing table structure comparison to generate a structure difference record, determining a data range according to a comparison mode parameter, executing data volume comparison to generate a quantity difference value, determining a data sampling strategy, and performing content comparison to generate a feature identifier comparison result; and finally, integrating the structure difference record, the quantity difference value and the feature identification comparison result to generate a verification report. According to the method, the data range and the sampling strategy are driven by comparing the mode parameters, so that the integrated processing of structure comparison, quantity comparison and content comparison is realized, and the accuracy and efficiency of data synchronization verification are improved in a heterogeneous database environment.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a database data verification method, device, equipment and storage medium. Background Art

[0002] During integration testing or system testing of data synchronization functions, data synchronization verification is a critical task, especially when it comes to data flow and structure mapping between multiple source systems. The accuracy and consistency of data synchronization directly affect the quality of subsequent data analysis and the stability of the business system. Traditional data synchronization testing methods rely heavily on manual verification of the structure and content of source and target data. Although intuitive, they lack significant manpower input, efficiency assurance, and the processing of complex data structures. When the synchronization object contains a large number of data tables, the data volume is large, or the structural differences are complex, manual testing is not only inefficient, but the verification results are easily interfered with by subjective factors, making it difficult to form a standardized and reusable verification process.

[0003] In the fintech business sector, data synchronization often involves synchronizing account data, transaction flows, customer profiles, and other information between heterogeneous systems. Because it involves sensitive data that changes frequently, financial systems have particularly stringent requirements for data consistency, field matching accuracy, and the trajectory of field value changes. However, existing testing processes often only focus on the number of records in the main field or spot checks on the content of some fields. There is a lack of a comprehensive visual verification mechanism for differences in database structure evolution, the legitimacy of field type mappings, and record-level differences. This can easily lead to data inconsistencies in key links such as fund settlement and risk control, creating systemic risks.

[0004] In the healthcare sector, data synchronization is common across systems, such as the exchange of medical records, diagnostic data, and medication records between in-hospital systems and regional medical platforms. In these scenarios, field standards, storage formats, and time field accuracy vary between systems, making traditional manual verification difficult to handle with complex structures, diverse fields, and frequently updated medical data. In large-scale batch synchronization or real-time update scenarios, relying on sampling for data verification can hardly detect marginal field differences or inconsistencies in specific records, which can easily lead to distorted clinical data references, deviations in medical order execution, and even trigger regulatory compliance risks.

[0005] Furthermore, in the current climate of agile software development and increasingly complex data systems, enterprises are facing the dual challenges of compressed testing cycles and high test quality. If data synchronization testing continues to rely primarily on manual spot checks, it will be unable to meet multi-dimensional quality assurance requirements, including structural comparison accuracy, data content consistency verification depth, and change scope tracking. This testing approach is insensitive to data structure changes and cannot effectively identify differences in field types, field lengths, or field semantic conversions. It also struggles to cope with the testing pace of high-frequency releases and multi-environment deployments.

[0006] Existing technologies lack an efficient, automated, and structured data consistency verification mechanism to fully support data synchronization testing requirements across complex systems. This is particularly true in scenarios involving heterogeneous structures, complex data, and multi-environment joint debugging. The system lacks the ability to systematically and closed-looply verify structural differences, data volume changes, and field content matching. Therefore, integrating a technical approach that supports multiple databases, detects structural differences, performs content-level sampling verification, and outputs structured results into the testing process remains an unresolved challenge in the field of data synchronization testing. Summary of the Invention

[0007] The main purpose of the present invention is to provide a database data verification method, device, equipment and storage medium, aiming to solve the technical problem in the existing technology that there is a lack of a unified data consistency verification mechanism driven by multi-mode parameters, and it is difficult to achieve comprehensive comparison at the structural, quantitative and content levels at the same time.

[0008] To achieve the above object, the present invention provides a database data verification method, comprising:

[0009] Configure the connection parameters of the source database and the target database;

[0010] Setting comparison task parameters and associating the comparison task parameters with connection parameters of a source database and a target database, wherein the comparison task parameters include a source table identifier, a target table identifier, and a comparison mode parameter;

[0011] Determine a source table corresponding to the source table identifier in the source database, and determine a target table corresponding to the target table identifier in the target database;

[0012] Performing a table structure comparison operation on the source table and the target table to generate a structure difference record;

[0013] Determine a data range according to a comparison mode parameter, perform a data quantity comparison operation on the source table and the target table based on the data range, and generate a quantity comparison difference;

[0014] Determine a data sampling strategy according to the comparison mode parameters, perform a data content comparison operation on the source table and the target table based on the sampling strategy, and generate a feature identification comparison result;

[0015] The structural difference records, quantity comparison differences and feature identification comparison results are integrated to generate a verification report.

[0016] Furthermore, to achieve the above-mentioned object, the present invention provides a database data verification device, comprising:

[0017] Database connection configuration module, used to configure the connection parameters of the source database and the target database;

[0018] A comparison task generation module is used to set comparison task parameters and associate the comparison task parameters with the connection parameters of the source database and the connection parameters of the target database. The comparison task parameters include a source table identifier, a target table identifier, and a comparison mode parameter.

[0019] A table location and analysis module is used to determine a source table in the source database corresponding to the source table identifier, and to determine a target table in the target database corresponding to the target table identifier;

[0020] A structure difference comparison module is used to perform a table structure comparison operation on the source table and the target table to generate a structure difference record;

[0021] A data volume comparison module is configured to determine a data range according to a comparison mode parameter, perform a data volume comparison operation on the source table and the target table based on the data range, and generate a quantity comparison difference;

[0022] A data content comparison module is configured to determine a data sampling strategy according to comparison mode parameters, perform a data content comparison operation on the source table and the target table based on the sampling strategy, and generate a feature identification comparison result;

[0023] The verification result generation module is used to integrate the structural difference records, quantity comparison differences and feature identification comparison results to generate a verification report.

[0024] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a database data verification program stored in the memory and executable on the processor, and the database data verification program, when executed by the processor, implements the steps of the database data verification method described above.

[0025] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a database data verification program is stored. When the database data verification program is executed by a processor, the steps of the database data verification method described above are implemented.

[0026] Beneficial effects: The present invention relates to the field of data processing technology and can be applied to business scenarios such as financial technology and medical health. A database data verification method is disclosed, including: configuring source database connection parameters and target database connection parameters, setting comparison task parameters including source table identifier, target table identifier and comparison mode parameters, and associating the parameters with corresponding database connection parameters, determining the source table and target table corresponding to the source table identifier and the target table identifier, performing table structure comparison to generate structure difference records, determining the data range according to the comparison mode parameters and performing data volume comparison to generate quantity difference, further determining the data sampling strategy and performing content comparison to generate feature identification comparison results, and finally integrating the structure difference records, quantity difference and feature identification comparison results to generate a verification report. The present invention drives the adaptive selection of data range and sampling strategy through comparison mode parameters, integrates the structure difference comparison, data volume comparison and content-level feature comparison processes into a unified process, and can realize multi-dimensional consistency verification in heterogeneous database scenarios, thereby improving the accuracy and automation of data synchronization testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0028] Figure 1 A schematic diagram of an application environment of a database data verification method according to an embodiment of the present invention;

[0029] Figure 2 This is a flow chart of an embodiment of a method for verifying database data according to the present invention;

[0030] Figure 3 This is a functional module diagram of a preferred embodiment of the database data verification device of the present invention;

[0031] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0032] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0034] The database data verification method provided by the embodiment of the present invention can be applied in Figure 1In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can configure the source database connection parameters and the target database connection parameters through the user terminal, set the comparison task parameters including the source table identifier, the target table identifier and the comparison mode parameters, and associate the parameters with the corresponding database connection parameters, determine the source table and the target table corresponding to the source table identifier and the target table identifier, perform table structure comparison to generate structure difference records, determine the data range according to the comparison mode parameters and perform data volume comparison to generate quantity difference values, further determine the data sampling strategy and perform content comparison to generate feature identification comparison results, and finally integrate the structure difference records, quantity difference values ​​and feature identification comparison results to generate a verification report. The present invention drives the adaptive selection of data range and sampling strategy through comparison mode parameters, integrates the structure difference comparison, data volume comparison and content-level feature comparison processes into a unified process, can realize multi-dimensional consistency verification in heterogeneous database scenarios, and improve the accuracy and automation of data synchronization testing. Among them, the user terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server terminal can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific examples.

[0035] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a database data verification method provided by the present invention. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0036] like Figure 2 As shown, the database data verification method proposed by the present invention includes the following steps:

[0037] S10, configuring the connection parameters of the source database and the target database;

[0038] In this embodiment, configuring the connection parameters of the source database and the target database is a basic preparatory operation for the data consistency verification process, wherein the connection parameters refer to a set of configuration information required for connecting to the database instance, mainly including the database type, host address, port number, user name, password, and the database name or instance identifier that may be included. Database types include but are not limited to relational databases such as MySQL, Oracle, and PostgreSQL, as well as non-relational databases such as MongoDB. During the technical execution process, the user first inputs the set of parameter information through a visual configuration interface or interface. The system needs to perform differentiated parsing of the protocol specifications and connection requirements of different types of databases, and on this basis, build standardized connection objects or connection descriptors to adapt to the interaction requirements with the database during the subsequent comparison task execution.

[0039] The host address, as network-level location information, indicates the physical or virtual host location of the database instance. It typically appears as an IP address or domain name. This information serves as the basis for network connectivity verification. Before performing connection verification, you need to use network probing tools such as ping or TCP-based port handshaking to verify host reachability. The port number identifies the listening entry point for the database service within the host and serves as the transport layer communication channel identifier. If the port is not open or blocked by security policy, the connection will fail. Therefore, the system needs to use a port scanning tool to verify that the port is connectable.

[0040] Username and password, as authentication information, are the core elements of the database access control mechanism. Different databases have different support for authentication protocols. For example, Oracle supports complex descriptors in the TNS format, while MySQL uses a three-way binding of username + password + database name. Therefore, after receiving the authentication information, the system needs to identify the database type, load the corresponding driver adapter module, verify whether the authentication information format matches the current database type, and then call the corresponding connection driver to test the connection attempt. If the connection is successful, the system will return a connection available status indicator and persist the connection configuration record for reuse when subsequent data comparison tasks are called. The above operations apply not only to the process of establishing the source database connection, but also to the configuration operations of the target database connection. The two maintain symmetry in the process structure, but need to be executed independently to ensure the bidirectional integrity of the data path.

[0041] After connection verification, the system stores the validated connection parameters in a structured format, typically within a configuration library and stored under a designated configuration node. Each set of connection parameters is assigned a unique configuration identifier and is associated with the comparison task parameters in subsequent tasks. This allows for rapid database session establishment during comparisons, avoiding repeated parsing and verification operations. To enhance system scalability and reusability, each set of connection parameter configurations can be bound to multiple task entities, allowing the same database connection to be shared across multiple tasks, improving resource utilization and reducing configuration redundancy.

[0042] With support for different database types, a unified connection configuration interface module can be used. This module selects and loads the corresponding connection driver component based on the database type. For example, it loads the Oracle JDBC driver to parse the TNS descriptor and the MySQL driver to process the standard connection URL. It also encapsulates the connection test logic internally, including three stages: connectivity detection, port availability confirmation, and authentication information verification. For network connectivity verification, the ICMP protocol can be used to quickly determine the host response status. For port availability verification, tools such as Netty, Socket, or nmap can be called to perform TCP handshake operations. During the authentication and verification process, after constructing the connection string, an attempt is made to establish a minimum-privilege database session. If a valid session is returned, the connection is considered successful. The system records the connection test status and stores the parameter set in the configuration library in JSON format. In order to adapt to the proxy network, firewall policy, or springboard access mode in the enterprise environment, a connection preprocessing module can be added to perform rule mapping on the host address and port number or use a transit service to generate a connectable channel.

[0043] In multi-environment deployments, the configuration structure can be expanded to include environment tags (e.g., test, pre-release, and production) to mark the applicable environments for connection parameters. During task scheduling, matching connection parameter nodes are automatically selected based on the task environment type, enabling data source adaptation for the same logical task across different deployment environments. Furthermore, a connection parameter versioning mechanism can be added to save historical versions after connection parameter updates, supporting the need to reproduce and re-compare the connection environments of historical tasks.

[0044] Example description: In the medical and health data platform, the patient diagnosis and treatment records in the hospital HIS system are stored in the Oracle database, while the regional medical information platform uniformly uses MySQL as the data integration base. Testers need to configure Oracle as the source database and MySQL as the target database to perform data synchronization consistency testing. Through the above mechanism, the user enters the IP address, port number and authentication information of the HIS system in the platform interface. The system detects the TNS connection format of the Oracle database and successfully establishes a connection channel. At the same time, it configures the target database connection and completes the authentication verification. After the connection parameters are verified, the system binds the two into a set of comparison connection configuration items and marks them as "test environment". All subsequent comparison processes of HIS synchronization tasks in the test environment can call this connection configuration to ensure the continuity and accuracy of the test tasks.

[0045] In financial scenarios, the credit scoring engine's data source is synchronized and converged from multiple subsystems. The data synchronization process involves exchanging data between the accounting systems and data warehouses of different branches, some using PostgreSQL and others using MongoDB. During testing, connection parameters for multiple heterogeneous data sources are configured. The system loads the corresponding drivers and performs authentication tests on each set of connection information. Once verified, a structured configuration is formed. The data synchronization and comparison tasks corresponding to each scoring model can be linked to and called from the corresponding connection nodes, enabling dynamic adaptation of connection parameters across multiple source paths, ensuring that the data sources received by the scoring engine always come from legitimate, stable, and verified systems.

[0046] By modularizing and structuring database connection parameter configuration and introducing a three-layer verification mechanism for network connectivity, port status, and authentication compatibility, invalid connections can be effectively filtered before the comparison task begins, avoiding task execution interruptions and improving system operational stability. By configuring the binding method between nodes and task parameters, tasks and connection configurations are decoupled, improving configuration reusability and management efficiency. At the same time, through structured storage of connection parameters and environmental identification control, unified access control and dynamic adaptation across multiple environments and databases are supported, improving the flexibility and accuracy of the entire data consistency verification system in multi-system docking scenarios.

[0047] S20, setting comparison task parameters, and associating the comparison task parameters with connection parameters of the source database and the target database, wherein the comparison task parameters include a source table identifier, a target table identifier, and a comparison mode parameter;

[0048] In this embodiment, setting the comparison task parameters is a key initialization operation after the connection configuration is completed. The comparison task parameters are a set of core metadata used to control the data comparison behavior, which determines the target scope of the comparison task, the comparison method and its binding relationship with the connection resources. Among them, the source table identifier and the target table identifier are used to uniquely identify the data objects to be compared in the source database and the target database, usually consisting of a library name, a table name or a combination thereof, and may also include multi-level information such as a schema name, an alias or a namespace. As the starting point of the comparison task, the correctness of the table identifier directly determines whether the comparison operation can accurately act on the expected data table object. Therefore, during the setting process, the table identifier needs to be semantically parsed and standardized, such as unifying uppercase and lowercase letters, removing prefixes and suffixes, and handling separator compatibility (such as replacing periods with slashes).

[0049] The comparison mode parameter is used to control the execution logic of the comparison task. It defines the task's strategic path in terms of data volume and data content verification. Common comparison modes include full comparison mode and incremental comparison mode. Full comparison is suitable for scenarios such as the first synchronization and historical data verification. The system will compare all records in the source table and the target table; incremental comparison is suitable for regular synchronization and update task result verification. The system only compares data subsets within a specified time period or under conditions. This parameter not only affects the data range determination logic of subsequent comparisons, but also directly determines the sampling strategy and matching depth adopted in the data content comparison process. Therefore, it needs to be explicitly set as the basic meta-attribute of the task and maintained in a stable manner throughout the life cycle of the task.

[0050] When setting the comparison task parameters, you also need to bind them to the source and target database connection parameters of the established connection channels. The binding mechanism maps the table identifier fields in the task parameter structure to the corresponding connection nodes in the connection configuration library, allowing the comparison task to directly call the verified connection channel to complete data access and reading operations at runtime. This binding behavior ensures that the connection context of the comparison task is clear, avoiding resource conflicts, connection confusion, or abnormal access permissions at runtime. This process is usually completed in the platform-level task setting interface or by uploading a batch template file. The system parses the file content and automatically completes the task parameter generation and binding operations.

[0051] Task parameter settings must support a variety of input paths, including manual interactive input and template import. The interactive input method is suitable for configuring a single table or a small number of tables one by one. It supports filling in the source table identifier and target table identifier, and selecting the comparison mode through a form. Once the selection is made, the corresponding connection configuration items are automatically linked for user confirmation. The template import method is suitable for batch configuration of multiple comparison tasks. The system parses the data of each task row in the template and identifies the corresponding connection configuration identifier field to complete parameter construction and resource binding, achieving efficient task initialization.

[0052] For interactive task configuration, the platform's front-end interface provides a multi-field input area where users select the source connection configuration, target connection configuration, source table identifier, target table identifier, and comparison mode. After receiving the input, the system backend preprocesses the table identifier, including removing illegal characters, standardizing delimiter formats, and regularizing field name formats. The system then verifies the table identifier against a verified database connection node to ensure that the entered table actually exists in the target database instance. If the verification passes, the system constructs the task parameter structure and stores it in the task configuration center.

[0053] For template import, the platform supports uploading files in Excel or CSV format. Each line in the file represents a comparison task and contains fields such as source connection identifier, target connection identifier, source table identifier, target table identifier, and comparison mode. After the system parses the file, it automatically queries the corresponding connection node in the configuration library based on the connection identifier, completes the batch generation of comparison task parameters, and uniformly performs parameter validity verification and binding storage operations. For table name fields, it supports pre-processing operations such as batch prefix replacement and unified naming mapping to improve configuration efficiency in large database environments.

[0054] To accommodate multiple database types or multi-environment deployments, the task parameter setup logic can also automatically map input table identifiers to paths based on the environment tags bound in the connection configuration. For example, when a test task specifies the "pre-release" source environment, the system will automatically map "user_data" to "pre_user_data" and complete the parameter injection operation. In financial industry applications, data domain identifiers (such as account domains and transaction domains) can also be introduced to organize tasks and bind resources by data domain, further improving the clarity and manageability of task structures.

[0055] Example: In the medical and health data supervision platform, testers need to compare the prescription information received from the primary diagnosis and treatment system with the data reported to the regional platform for consistency. The source table is "prescription_record" in the community health system, and the target table is "rhc_prescriptions" in the regional data center. The task configurator selects the configured source system connection item and target platform connection item in the platform interface, enters the two table identifiers mentioned above, and selects the incremental comparison mode to limit the comparison to the data of the most recent week. The system verifies the legitimacy of the table identifier and completes the connection binding, successfully generates the task parameter item and stores it.

[0056] In financial risk data synchronization scenarios, multiple business subsystems need to synchronize real-time transaction information to a centralized risk control warehouse. Testers use batch template import to upload configuration files for multiple tables, such as transactions, reconciliation, and exception monitoring. The system parses the templates and automatically binds the corresponding connection parameters to each subtask, generating a batch comparison task parameter structure and verifying the binding. This approach significantly shortens task initialization time and improves cross-system configuration accuracy, meeting the financial industry's practical needs for efficient multi-task management and batch testing.

[0057] By setting comparison task parameters and binding them to database connection configurations, we can effectively model the association between comparison targets and data channels, ensuring that comparison tasks have complete data access context during execution. The standardized task parameter structure and support for multiple input methods provide the system with flexible task configuration capabilities, meeting both single-shot fine-grained configuration requirements and large-scale batch initialization scenarios. This significantly improves task configuration efficiency, reduces configuration error rates, and provides a stable and reliable foundation for subsequent comparison processes.

[0058] S30, determining a source table corresponding to the source table identifier in the source database, and determining a target table corresponding to the target table identifier in the target database;

[0059] In this embodiment, determining the source table corresponding to the source table identifier in the source database and the target table corresponding to the target table identifier in the target database is a key path parsing operation after the connection configuration and task parameter setting in the data consistency verification process. The core purpose of this operation is to map the abstract table identifier to an actually accessible physical table object in the database, so that the system can accurately perform subsequent comparison operations on the structure, data volume, and data content based on the mapping result. The source table identifier and the target table identifier are input parameters, usually expressed in the form of strings, which may include multiple semantic units such as database name, schema name, table name, view name, or namespace prefix.

[0060] During the specific execution process, the input table identifier must first be semantically parsed, including extracting the naming hierarchy based on the database type and performing unified standardization. During the standardization process, the system performs format conversion operations based on the naming conventions of different databases. For example, the system performs name unification for case-sensitive databases, parses the "library.table" format into a hierarchical path structure for databases that support multi-level naming, and replaces special characters in specific databases (such as the "." used in MongoDB) with symbols to generate a standard table identifier that conforms to the current database access rules. This standard table identifier is the basis for subsequent metadata queries and table object matching.

[0061] After standardizing table identifiers, the system calls the corresponding metadata access interfaces in the source and target databases. These interfaces are typically accessed through standard SQL meta-commands or database-specific system tables, and are used to query whether data objects matching the standard table identifier exist in the current database. For relational databases such as MySQL, PostgreSQL, or Oracle, table existence verification can be performed by accessing metadata repositories such as information_schema and ALL_TABLES, and metadata such as table structure and field definitions can be extracted. For non-relational databases such as MongoDB, collection existence and document structure information can be confirmed by executing the "showcollections" command or calling the system collection interface.

[0062] This operation also requires a certain level of fault tolerance and compatibility processing logic, especially in heterogeneous system integration scenarios, where actual table names and table identifiers may differ due to inconsistent naming conventions, naming conflicts, or middleware renaming. The system must have a built-in naming fuzzy matching mechanism or field alias matching strategy, for example, supporting fuzzy comparison of "emp_no" and "empno" as possible synonyms, or establishing a mapping dictionary for unified field name recognition across multiple platforms to improve target table recognition accuracy.

[0063] After completing the identification of the source table and the target table, the system will return a structured table object information structure, including information such as the table name, field name list, field type, field length constraint, etc., and store it in the context task cache or the data channel of the table structure comparison module for direct reference by subsequent comparison tasks.

[0064] For example, in an Oracle database scenario, if the source database is Oracle and the user sets the source table identifier to "CRM.customer_info," the system parses the identifier and recognizes that it contains the database schema name "CRM" and the table name "customer_info." Furthermore, Oracle is a case-sensitive database and the table name must be standardized to uppercase "CUSTOMER_INFO." A query is then constructed to access ALL_TABLES and ALL_TAB_COLUMNS to obtain table structure information. If the target database is MySQL and the target table identifier is "customer_info," the system does not require case conversion, but requires a unified naming format to exclude special characters. The system then verifies its existence by querying information_schema.tables to extract field metadata.

[0065] In a MongoDB scenario, the source table might be identified as "user.profile_info." After parsing, the system identifies "user" as the database name and "profile_info" as the collection name. It then accesses the collection list through the MongoDB driver to confirm the collection's existence and obtains a sample document structure for subsequent field extraction. The system can also configure a collection alias mapping table to establish a mapping between possible naming differences, such as "profile_info" and "profileinfo," enabling automatic collection identification and fault tolerance.

[0066] When deployed across environments, the system supports an environment-aware table identification mechanism. During the identification phase, the system applies environment prefix or suffix mapping to table identifiers based on the environment tags configured in the connection. For example, "user_data" in the test environment is automatically mapped to "test_user_data" to improve the environment consistency of task configuration and data access.

[0067] Through the combined implementation of standardized processing and database metadata access interfaces, we achieve a precise mapping of abstract table identifiers to actual data table objects, establishing the data starting and end points for the comparison task. This operation incorporates multiple strategies, including identifying database type differences, unifying naming standards, fault-tolerant identification, and extracting field structures. This improves the accuracy, stability, and adaptability of table identification across systems and heterogeneous database environments, providing a reliable data foundation for subsequent structural comparisons, data volume comparisons, and data content verification.

[0068] S40, performing a table structure comparison operation on the source table and the target table to generate a structure difference record;

[0069] In this embodiment, performing a table structure comparison operation between the source table and the target table is a core step in the data consistency verification process for determining whether the data structure is consistent. The object on which the structural comparison is based is the standardized table structure information obtained by parsing the source table identifier and the target table identifier in the previous step, including key attributes such as field name, field data type, field length constraint, and field order. The fundamental purpose of the comparison process is to identify inconsistencies between the two tables at the structural level, including but not limited to differences in the number of fields, differences in field names, incompatible field types, and inconsistent field length definitions.

[0070] This operation first extracts the field name list, data type definition and length constraints from the source table structure information, and constructs them into a sequence of structure tuples, for example, in the form of (field name, type, length) for standard expression. The same operation is performed synchronously on the target table structure to generate a target-side structure tuple sequence. During the structural comparison process, the system constructs a field name mapping relationship based on the field name matching strategy. The field name matching method usually includes two mechanisms: exact matching and fuzzy matching. Exact matching is used to locate fields with exactly the same field names, while fuzzy matching is suitable for scenarios with common naming differences, such as the matching of "cust_id" and "customer_id". Approximate matching of field semantics is achieved in the system by configuring alias mapping rules or editing distance calculation.

[0071] For matched field pairs, the system further checks the compatibility between the source and target field types. This compatibility check is based on the database type mapping table, which defines the appropriate path for type conversion between different databases. For example, the INT type in MySQL is compatible with the NUMBER type in Oracle, and VARCHAR and TEXT can be roughly compatible based on length thresholds. If a conversion path exists for a field type, the system records the field as "compatible type"; otherwise, it is marked as "incompatible type."

[0072] Field length comparison logic determines whether the permitted lengths of source table fields are consistent or compatible with those of target table fields. For character fields (such as VARCHAR and NVARCHAR), if the maximum length defined for the target field is less than that of the source field, this is considered a potential truncation risk and the system will mark the field as "length incompatible" and record the source and target lengths, along with the difference. For numeric fields, the precision of the integer and decimal parts may be further split and compared to improve risk identification accuracy.

[0073] After all fields are compared, the system generates a structured difference record based on the comparison results. This record is output in a structured format. Each record contains information such as the field name, comparison type (such as missing field, incompatible type, inconsistent length), inconsistency location (source or target), and recommended repair strategy. The structure difference record is stored in the difference analysis module for subsequent report generation and operation and maintenance processing.

[0074] Example: In the implementation of a healthcare data platform, the community health system uses a PostgreSQL database, with the field "patient_age" defined as INTEGER. The provincial data platform uses a MySQL system, with the corresponding field "age" defined as TINYINT. During a structural comparison, the system identified a high degree of similarity in the field names and performed a fuzzy match, determining that they were synonymous fields. Furthermore, the system identified weak compatibility between the two types and detected that the target field length was insufficient to accommodate the numerical range of some elderly patients. Therefore, the field was marked as "length incompatible" and a prompt message was generated suggesting a length change.

[0075] In a financial data integration scenario, the field "txn_amt" in the source table "branch_txn_detail" is defined as DECIMAL(16,4), while the target data warehouse uses the FLOAT type. The system identifies this type conversion as unsafe and may result in precision loss. It records the structural difference as "Type Incompatibility" and prompts the target system to adjust the field type or precision definition to ensure the accuracy of the transaction amount field during financial calculations after synchronization.

[0076] Structural comparison can proactively identify structural inconsistencies caused by database design differences or synchronization rule changes, preventing risks such as missing fields, type conflicts, and data truncation during data content comparison or synchronization. Comparing the front-end structure provides a basis for optimizing comparison strategies, adjusting field mapping rules, and pre-synchronizing and repairing target tables, improving the stability and predictability of the entire data consistency verification process.

[0077] S50, determining a data range according to a comparison mode parameter, performing a data volume comparison operation on the source table and the target table based on the data range, and generating a quantity comparison difference;

[0078] In this embodiment, determining the data range based on the comparison mode parameters and performing data volume comparison accordingly is a key operation for identifying differences in the number of records during the data consistency verification process. The comparison mode parameters serve as the core control input and determine the range of the record set for subsequent data volume comparison. This parameter mainly includes two situations: full mode and incremental mode. The corresponding technical processing paths, adaptation scenarios, and comparison logic are significantly different. Therefore, the system needs to logically define the data range based on the comparison mode parameters before executing this step.

[0079] When the comparison mode parameter is set to Full, the data scope is defined as the set of all existing records in the source and target tables. In this mode, the system does not introduce any filtering conditions and directly counts the total number of records in both tables to obtain the total source and target data. This processing logic is suitable for scenarios such as initial data synchronization verification and periodic full-database consistency checks. Its core purpose is to evaluate the completeness of the overall data replication or migration process.

[0080] When the comparison mode parameter is incremental mode, the system needs to parse the filter expression from the comparison task parameters and build the data range filtering logic. The filter expression usually includes time field range (such as "update time is greater than a certain time point"), status field restriction (such as "record status is 'active'") or compound logic (such as "status is active and time is within the last seven days"). The system injects the expression into the query statements of the source table and the target table to obtain the incremental record set of the source table and the incremental record set of the target table respectively. This mode is suitable for real-time data synchronization or continuous small batch write scenarios, and is used to track data consistency during the incremental update process.

[0081] The data volume comparison operation is essentially a count comparison based on the data range. The system counts the number of source table record sets as the source data quantity and the number of target table record sets as the target data quantity. In full-volume mode, the system uses absolute difference logic, that is, taking the absolute value of the difference between the source data quantity and the target data quantity to generate a full-volume quantity comparison difference, which is used to reflect whether the overall record count is consistent. In incremental mode, symbolic difference logic is used, that is, subtracting the target incremental quantity from the source incremental quantity to generate a quantity comparison difference with positive and negative signs, thereby indicating directional differences in the record synchronization process, such as the source table having more than the target table, or the target table having more than the source table.

[0082] After generating the quantity comparison difference, the system records it in a structured format, which can be used for subsequent verification report presentation or anomaly flagging. If the quantity difference exceeds a preset threshold, the system triggers an alert module, marking the comparison task as failed for further analysis.

[0083] For example, in full-data mode, the system executes unconditional record count statements through a database connection. For example, execute "SELECT COUNT(*) FROM source_table" on a MySQL source table and perform the same statistics on the target table to obtain the complete record count. The system then calculates the absolute difference between the two and outputs it as the full-data count comparison difference.

[0084] In incremental mode, the system first extracts a filter expression from the comparison task parameters, such as "last_updated > = '2023-01-01'." It then constructs a qualified query on both the source and target tables, for example, "SELECT COUNT(*) FROM source_table WHERE last_updated > = '2023-01-01'." The resulting counts are used as the source and target increments, respectively. The system then directly subtracts the two, preserving the sign of the difference to create a symbolic increment difference.

[0085] To enhance system compatibility, the definition of filtering condition templates and time placeholder parsing functions can be supported during the task configuration phase. Dynamic injection of parameters such as "{today-1}" is supported, and the specific time range corresponding to the current runtime context is automatically replaced during task execution, thereby improving the automation level of incremental comparison.

[0086] By controlling the data range in conjunction with comparison mode parameters, adaptive adjustment of the comparison strategy is achieved in different verification scenarios. Full mode provides a comprehensive assessment of overall record consistency, while incremental mode supports consistency monitoring of changed data. The difference generation mechanism not only determines the existence of quantitative differences, but also reflects their extent and direction, providing a guide for subsequent data content comparison or anomaly repair, thereby significantly improving the accuracy and responsiveness of data synchronization verification.

[0087] S60, determining a data sampling strategy according to the comparison mode parameters, performing a data content comparison operation on the source table and the target table based on the sampling strategy, and generating a feature identification comparison result;

[0088] In this embodiment, determining a data sampling strategy based on comparison mode parameters and performing a data content comparison on the source and target tables based on this strategy is the core process for implementing field-level data consistency verification. Its primary purpose is to perform a refined comparison of field values ​​based on structural and quantitative verification to identify micro-differences during data synchronization. The existence of comparison mode parameters determines the data sampling mechanism used by the system during data content verification, thereby influencing subsequent data extraction methods, field summarization methods, and comparison granularity.

[0089] When the comparison mode parameter is full mode, the system needs to process all records in the table to achieve comprehensive field value verification. In this scenario, the sampling strategy is defined as a shard processing strategy. This strategy logically divides the source table and the target table according to a key field (such as the primary key) to form several data shards. The record set of each data shard will be extracted separately, and the system will generate corresponding field value summary information for the field value of each record. Common methods include standardizing the complete record sequence and generating MD5, SHA256 or other hash identifiers to obtain the unique content feature identifier of the record. The system then scans the target table in units of shards to see if there is a field value summary that is consistent with the corresponding shard of the source table. If so, the record is successfully matched. If not, a difference entry is recorded. The comparison results of each shard will be stored and marked independently, and the comparison results of all shards will eventually be integrated to form a complete full-scale feature identification comparison result.

[0090] When the comparison mode parameter is incremental mode, the system needs to perform sampling processing on the determined source table incremental record set. The sampling strategy is a specific form of data sampling strategy. Its core purpose is to dynamically calculate the sampling ratio based on the scale and characteristics of the source table incremental data, so as to balance the verification coverage and performance efficiency in incremental scenarios with large data volume and frequent changes. The determination of the sampling ratio is usually based on the data volume of the source table incremental record set, combined with the preset ratio mapping table or configuration parameters for deduction. For example, when the number of records is below a certain threshold, a high-proportion sampling is used, and when the number is too large, an exponential decay ratio strategy is used. After determining the sampling ratio, the system extracts some records from the source table incremental record set to form a sampled record set, and then generates summary information for the field values ​​of each record in the set, matches them one by one with the records in the target table, and records difference indicators such as the content of unmatched fields and time field deviation.

[0091] The feature identification comparison results, composed of the output of the aforementioned matching operations, include statistics and details on successfully matched records, records with inconsistent field values, and missing records. This result not only supports the generation of content-level comparison reports but also can be used to analyze data modification errors, data update delays, or write losses within the synchronization link, providing a crucial guarantee for enhancing the depth and breadth of data consistency verification.

[0092] For example, in full-scale mode, the system first logically shards the table data based on the primary key field. The table is divided into several subsets based on the primary key value range or hash value interval, such as one shard for every 100,000 records. After the sharding is completed, the system reads the records of each shard, standardizes the field values ​​of each record (removing spaces, unifying upper and lower case, etc.), converts it into a JSON string and generates an MD5 digest to obtain the characteristic identifier of the record. The system then uses these identifier values ​​to search for matching values ​​in the corresponding shard of the target table. If the match fails, the data content is considered inconsistent.

[0093] In incremental mode, the system first invokes the time field filtering logic, extracting filter conditions such as "update_time >= '2024-01-01'," and then constructs an SQL statement to retrieve incremental records. If the record set contains 30,000 records, the system uses the configured mapping table to determine a 5% sampling ratio, randomly selecting 1,500 records. Field value summaries are generated for these records, and matching operations are performed on the incremental record set in the target table to identify field value deviations. Specifically, a deviation window comparison is performed on time fields, recording the deviation in seconds or a date error label.

[0094] All records that fail to match will form a sample content difference record set. When the system outputs the feature identification comparison results, the results of the full and incremental scenarios will be stored in a unified format for use by downstream reporting modules and alarm systems.

[0095] Through the adaptive selection of sampling strategies and the precise matching mechanism of data content summaries, not only the accuracy of field-level verification is improved, but also performance is guaranteed when dealing with large-scale data. Sharding processing in full-volume scenarios can effectively parallelize data scanning and comparison, which is suitable for deep data consistency verification needs; the dynamic sampling mechanism in incremental scenarios can adjust resource investment based on the actual data scale, achieving a balance between coverage efficiency and execution cost. This mechanism also eliminates the interference caused by format differences between databases on content matching through field value summaries, improving the stability and adaptability of cross-platform consistency detection.

[0096] S70, integrating the structural difference records, quantity comparison differences and feature identification comparison results to generate a verification report.

[0097] In this embodiment, integrating the structural difference records, quantity comparison differences and feature identification comparison results to generate a verification report is the final step of the data consistency verification process. Its function is to structurally integrate and uniformly express the outputs of various previous comparison operations, and provide standardized result support for manual review, system evaluation or automated linkage. This process is not just about splicing and presenting the comparison results, but also includes multiple technical sub-links such as multi-dimensional result aggregation, indicator normalization processing, result status judgment and output content formatting. The generated verification report not only needs to have data accuracy, but also should have structural clarity, problem location capabilities and output format requirements that adapt to different receiving systems or roles.

[0098] Structural difference records are field-level structural inconsistency information extracted during the table structure comparison process, including information such as field name, field data type difference, field length difference, and field missing status. Its internal structure is usually a set of difference items in the field dimension. The quantity comparison difference comes from the result obtained when comparing the number of data records, which is manifested as the difference in the number of records between the source table and the target table, including two types: full absolute difference and incremental symbolic difference. The feature identification comparison result is a content comparison product based on the field value summary, specifically including unmatched field value difference entries, time field deviation, sharding or sampling matching result status, etc., and has the function of determining data content-level differences.

[0099] When integrating these heterogeneous comparison results, the system must implement a unified data model mapping. This model, based on the core structure of "Comparison Dimension → Difference Type → Difference Details," establishes consistent representation of structural, quantity, and content comparison results across the field dimension and logical hierarchy. For example, if the field "amount" is identified as type-incompatible at the structural level, has no impact on quantity comparison, and exhibits decimal precision errors in content comparison, the final report should integrate these three levels of differences under this field and provide recommendations.

[0100] Furthermore, when generating reports, the system generates graphical or numerical output formats based on pre-set output templates. The report should display structural, quantitative, and content differences by dimension, assigning severity labels to each difference (e.g., fatal, warning, recommendation), and calculating the overall comparison result status (e.g., passed, failed, manual review required). Field-level differences are highlighted, enabling quick navigation and problem location.

[0101] The final verification report can be output in multiple formats, including web-based interactive views, JSON structured result files, PDF text reports, CSV difference details tables, etc., for subsequent processing by front-end interfaces, scheduling systems, or other data quality monitoring platforms.

[0102] By integrating comparison results across three levels—structure, quantity, and content—we can comprehensively verify the integrity, consistency, and accuracy of the data synchronization or migration process, resolving technical challenges inherent in existing methods, such as fragmented comparison results, difficulty locating issues, and ambiguous reporting granularity. Structured report output not only improves manual review efficiency but also provides an executable technical basis for subsequent automated alerting, quality scoring, synchronization rollback, and re-recording decisions, ensuring excellent system connectivity and platform adaptability.

[0103] The present invention relates to the field of data processing technology and can be applied to business scenarios such as financial technology and medical health. A database data verification method is disclosed, comprising: configuring source database connection parameters and target database connection parameters, setting comparison task parameters including source table identifier, target table identifier and comparison mode parameters, and associating the parameters with corresponding database connection parameters, determining the source table and target table corresponding to the source table identifier and the target table identifier, performing table structure comparison to generate structure difference records, determining the data range according to the comparison mode parameters and performing data quantity comparison to generate quantity difference, further determining the data sampling strategy and performing content comparison to generate feature identification comparison results, and finally integrating the structure difference records, quantity difference and feature identification comparison results to generate a verification report. The present invention drives the adaptive selection of data range and sampling strategy through comparison mode parameters, integrates the structural difference comparison, data quantity comparison and content-level feature comparison processes into a unified process, can realize multi-dimensional consistency verification in heterogeneous database scenarios, and improves the accuracy and automation of data synchronization testing.

[0104] In one embodiment, the above step S20 includes:

[0105] S201, receiving a source table identifier and a target table identifier input through an interactive interface, or receiving an uploaded comparison template file;

[0106] S202, when receiving a source table identifier and a target table identifier through the interactive interface, verifying the existence of the source table identifier and the target table identifier in the target database, and receiving comparison mode parameters corresponding to the verified source table identifier and the target table identifier;

[0107] S203, when receiving the uploaded comparison template file, parsing the comparison template file to obtain the source table identifier, the target table identifier, and the associated comparison mode parameters;

[0108] S204, generating comparison task parameters including a source table identifier, a target table identifier, and comparison mode parameters;

[0109] S205 , respectively associating the source table identifier in the comparison task parameters with the connection parameters of the source database, and the target table identifier with the connection parameters of the target database.

[0110] In this embodiment, setting the comparison task parameters is the bridge link between the input configuration and the subsequent comparison logic in the entire data consistency verification process. Its core lies in generating an information set describing the data comparison unit in a standardized manner and establishing a mapping relationship between it and the database connection configuration. The comparison task parameters contain three core components: the source table identifier, the target table identifier, and the comparison mode parameter. The source table identifier and the target table identifier are used to calibrate the data table entities to be compared. The comparison mode parameter is used to indicate the comparison strategy that should be adopted for subsequent comparison operations, such as full comparison or incremental comparison.

[0111] During the task configuration process, the system supports two data access methods. The first method is manual input through the interactive interface, where the user enters the source table identifier and the target table identifier in the front-end interface. This type of input usually contains the logical path within the database, such as database name.table name or schema name.table name. After receiving these identifiers, the system will call the metadata verification module to access the source database and target database respectively to confirm that the tables corresponding to these identifiers do exist in the connected database and that their structural information can be accessed. This verification operation is usually implemented by querying the database system table or calling the information model interface (such as MySQL's information_schema and Oracle's DBA_TABLES). After the verification is passed, the system allows the user to specify the comparison mode parameter, which exists in the form of an enumeration value (such as FULL for full amount, INCREMENTAL for increment), which is used for subsequent comparison logic branch selection.

[0112] The second method is to batch upload a comparison template file. Users can complete the input of multiple comparison task parameters by uploading a configuration file, such as an Excel file. Each line in the template file represents a comparison task and contains fields such as the source table identifier, target table identifier, and comparison mode. The system parses the file, verifies the correctness of the field format, extracts the data structure of each comparison task unit, and completes the batch task configuration. This path is suitable for large-scale multi-table data synchronization verification scenarios and significantly improves configuration efficiency.

[0113] After the task parameters are constructed, the system needs to establish a binding relationship between them and the connection parameter set. This process involves parsing the abstract table identifier into a mapping operation for the actual connection configuration. The system matches the source database connection parameters configured in the previous step with the target database connection parameters using the table identifier field in the task parameters to establish a directional mapping. For example, if the source table identifier is "db1.schema1.tableA," the connection parameters matching "db1" in the database configuration are automatically associated. This operation ensures that the task can correctly obtain the data connection context when performing the comparison, achieving a two-way binding relationship between the comparison task parameters and the database connection, and forming a complete and executable task entity.

[0114] Example: In building a regional healthcare platform, the electronic medical record (EHR) table "ehr_clinical_notes" needs to be regularly synchronized from various hospitals to a unified data warehouse. Using a template for batch upload, users can configure the table paths and target paths for 30 hospitals at once and set the comparison mode to "incremental." The system automatically verifies the existence of all tables, matches connection parameters, and generates 30 comparison task configuration items, supporting unified scheduling and report archiving.

[0115] In a financial scenario, a bank's data governance department needs to perform a full consistency check on the "account_transaction" table in five systems. The source and target table paths are entered one by one through an interactive interface. After successful real-time verification, the system generates task parameters and allows users to submit them to the scheduling queue with one click, triggering the subsequent automatic comparison process of structure, quantity, and content, effectively improving the efficiency of cross-system data consistency audits.

[0116] This implementation significantly improves the configuration flexibility, scalability, and execution accuracy of data comparison tasks by supporting interactive and template-based task configuration and introducing an automatic binding mechanism for comparison task parameters and database connection parameters. The task configuration process supports real-time verification, preventing connection failures or data capture anomalies caused by identification errors in the subsequent comparison phase. The unified task parameter structure also improves the consistency of data interfaces across the task scheduling system, reporting module, and alarm system, enhancing the overall system's engineering robustness.

[0117] In one embodiment, the above step S30 includes:

[0118] S301, parsing the naming hierarchy of the source table identifier, separating the source database name and the source table name, and generating a standardized source table identifier by case conversion and separator standardization;

[0119] S302, querying the metadata storage system of the source database to obtain source table structure information corresponding to the standardized source table identifier;

[0120] S303, parsing the naming format of the target table identifier, identifying special characters in the target table identifier that are incompatible with the target database type, replacing the special characters in the target table identifier with underscores, and generating a compatible target table identifier;

[0121] S304: Calling the metadata access interface of the target database to obtain target table structure information corresponding to the compatibility target table identifier.

[0122] In this embodiment, after the comparison task parameters are determined, the physical location and structural information of the source and target tables in their respective databases must be accurately located to perform subsequent structural comparison, data volume statistics, and content verification. The essence of this step is to parse the source and target table identifiers and, in conjunction with the database's metadata system, obtain standardized table structure information for comparison. This process involves multiple sub-steps, including naming convention processing, database compatibility adaptation, and metadata access interface invocation, ensuring good executability and consistency across heterogeneous database environments.

[0123] The source table identifier is typically entered by the user through task parameters. Its format may include a multi-layered structure consisting of the database name, schema name, and table name. Common forms include "database.schema.table" or "schema.table." To unify subsequent processing logic, the system parses this identifier, extracts the source database name and source table name, and performs format standardization. Standardization includes unifying uppercase and lowercase letters, removing redundant delimiters, and standardizing spaces and special characters. This allows table paths across different databases and naming conventions to be uniformly mapped to internal standard identifiers.

[0124] After the standardized source table identifier is generated, the system calls the source database's metadata storage system to query the table structure information. This process uses the corresponding interface based on the database type, such as information_schema.TABLES and COLUMNS for MySQL, ALL_TAB_COLUMNS and ALL_TABLES for Oracle, and pg_catalog.pg_tables and pg_attribute for PostgreSQL. The system performs an exact match based on the standardized source table identifier, extracting structural metadata such as field name, field type, field length, and whether it is a primary key, providing a structural baseline for the subsequent comparison module.

[0125] The processing logic for target table identifiers is similar to that for source tables. However, because the target database may use different naming conventions or have character set restrictions on identifiers (such as not supporting uppercase or special characters), naming format adaptation is required. The system first parses the target table identifier, identifies special characters that may cause access anomalies (such as hyphens, periods, and Chinese symbols), replaces them with underscores, and adjusts the case to form a compatible target table identifier. This adaptation strategy is particularly important for cross-database comparisons, table alias management, and partitioned table abstraction, and can significantly improve the accuracy of cross-system comparisons.

[0126] After the compatibility target table is identified, the system calls the target database's metadata access interface to retrieve the target table's structure. This entire structure acquisition process must support mechanisms such as delayed connections, connection pool reuse, and multi-tenant metadata isolation to ensure stability and scalability in large-scale concurrent scenarios.

[0127] This embodiment introduces naming resolution, format standardization, and database compatibility adaptation mechanisms, enabling the system to stably identify target and source tables and accurately extract structural information in heterogeneous database environments with significant structural differences. This mechanism addresses the problem of structure acquisition failures caused by inconsistent table identifiers, non-uniform naming formats, and different character set restrictions, ensuring that the comparison process has good cross-platform adaptability and structural awareness in actual deployment. In particular, during the integration of multiple databases and multiple business systems, highly accurate metadata extraction can be achieved, providing strong structural support for data consistency verification.

[0128] In one embodiment, the above step S40 includes:

[0129] S401, extracting a source table field name list, source table field data type definition, and source table field length constraint from the source table;

[0130] S402, extracting a target table field name list, target table field data type definition, and target table field length constraint from the target table;

[0131] S403, comparing the quantity difference between the source table field name list and the target table field name list, and generating a field quantity difference record;

[0132] S404, identifying fields with the same names in the source table field name list and the target table field name list, and generating a set of fields with the same names;

[0133] S405: for each field in the set of fields with the same name, query a preset database type mapping table to determine the convertibility of the source table field data type to the target table field data type, and generate a field type conversion status record;

[0134] S406 , for each field in the set of fields with the same name, verify the compatibility of the source table field length constraint condition and the target table field length constraint condition, and generate a field length matching result record;

[0135] S407: Integrate the field quantity difference record, the field type conversion status record, and the field length matching result record to generate the structure difference record.

[0136] In this embodiment, performing a structural comparison on the source and target tables and generating a structural difference record is a fundamental step in evaluating the accuracy and compatibility of the structural layer in data consistency verification. This process not only focuses on the consistency of field names, but also deeply analyzes whether the data type of the field can be mapped compatibly between the source and target databases, and whether the field length meets the transmission requirements of business data integrity. This step is aimed at heterogeneous database scenarios and needs to fully consider the differences in data type representation, length definition strategies, field naming rules, and conversion tolerances between different database vendors.

[0137] During the structure comparison operation, the first step is to extract the field name list, field data type definitions, and field length constraints from the source table structure description. This structural information is typically obtained through the database metadata interface obtained in the previous step, such as the field name "amount," data type DECIMAL, and length constraint (18, 2). This extraction process includes not only the field definitions themselves but also optional metadata such as the field order, whether it is a primary key, and whether it is a non-nullable field, which is used for subsequent alignment and convertibility analysis.

[0138] Accordingly, the field information extracted from the target table structure should be aligned one-to-one with the source table structure, generating a list of target field names, data type definitions, and length constraints. At this point, the system must perform a preliminary assessment of name existence, field position consistency, and semantic similarity to avoid misjudgments due to structural asymmetry during subsequent comparisons.

[0139] The first step in a structural discrepancy check is to compare the lengths of the source and target field lists to identify missing, redundant, or unusual fields. This process is used to identify significant structural inconsistencies. For example, if the source table has 15 fields and the target table has only 12, three fields will be marked as "missing" in the field count discrepancy record, providing a field-level warning.

[0140] Subsequently, the system identifies fields with exactly the same names and forms a set of fields with the same name. This set becomes the main object of structural compatibility verification. For each field in the set, the system queries the preset database type mapping table. This mapping table contains the correspondence between field types of various databases, such as Oracle's NUMBER can be mapped to MySQL's DECIMAL, PostgreSQL's TEXT can accept VARCHAR, etc. By matching the source field type with the target field type, it is determined whether it is in the convertible set, and a field type conversion status record is generated, marked as "compatible", "convertible", "incompatible" and other states.

[0141] In addition to type determination, field length compatibility is another core consideration. The system compares the length limits of identically named fields in the source and target tables. For example, if a source field is VARCHAR(100) and a target field is VARCHAR(50), the field is considered incompatible. If the target length is greater than or equal to the source length, the field is considered compatible. This determination is recorded as a field length matching result to assist in determining whether to increase the target field length or split the field design.

[0142] Finally, the system integrates the field quantity discrepancy records, field type conversion status records, and field length matching result records to construct a structural difference record object. This object lists field comparison items and inconsistencies, supporting structural layer visualization, highlighting, and problem remediation suggestions.

[0143] Example: In a healthcare data platform, a hospital synchronizes patient examination records from the HIS system to a regional data center. In the source table structure, the "report_time" field is defined as a DATETIME type, and the "exam_result" field is defined as a VARCHAR(1024). In the target database, however, the "report_time" field is a VARCHAR(50) and the "exam_result" field is a VARCHAR(512). When performing a structure comparison, the system determines through the type mapping table that there is an irreversible risk in converting from DATETIME to VARCHAR, and generates a field type conversion status record as "incompatible." Simultaneously, the system identifies that the "exam_result" field length is shorter than the source table field, and the field length matching result record is marked as "incompatible." Finally, the differences between these two fields are highlighted in the structure difference record, providing a constraint basis for subsequent data content comparison and preventing patient examination result information from being truncated due to insufficient target field length.

[0144] In the financial sector, a bank synchronized the loan transaction table "loan_txn" from its core system to the risk control platform database. During the structural comparison process, it was discovered that the target table's field "loan_status," while having the same name, was defined as an INT type on the target side, while it was an ENUM type on the source side. Based on pre-set mapping rules, the system determined that this type could not be directly converted and marked the field type conversion status as "Conversion Rejected." The system also discovered that the source table contained a field called "risk_level," which was completely missing from the target table. A field quantity discrepancy record indicated that this field was unique to the source side, prompting the target system to either add this field or set a default value policy to ensure structural consistency and risk control data integrity.

[0145] This embodiment uses a three-dimensional comparison of field name, data type, and length to proactively identify potential data transmission mismatches, field truncation, or type conversion failures at the structural level, preventing structural vulnerabilities from being exposed until later in data synchronization. Structural difference records provide a granular, field-level basis for judgment, enabling problem location, risk assessment, and implementation, significantly improving the pre-processing capabilities and accuracy of data synchronization and consistency verification.

[0146] In one embodiment, the above step S50 includes:

[0147] S501, when the comparison mode parameter is the full mode, determining the data range to be the full record set of the source table and the full record set of the target table;

[0148] S502, counting the total number of records in the full record set of the source table and the total number of records in the full record set of the target table, and generating a total source data volume and a total target data volume respectively;

[0149] S503, determining the absolute difference between the total amount of source data and the total amount of target data, and generating a total amount comparison difference;

[0150] S504, when the comparison mode parameter is the incremental mode, parsing the filter condition expression in the comparison task parameter;

[0151] S505, determining that the data range is a source table incremental record set consisting of records in the source table that meet the screening condition expression, and a target table incremental record set consisting of records in the target table that meet the screening condition expression;

[0152] S506, counting the number of records in the source table incremental record set and the number of records in the target table incremental record set, and generating a source incremental quantity and a target incremental quantity respectively;

[0153] S507, determining a symbolic difference between the source incremental quantity and the target incremental quantity, and generating an incremental quantity comparison difference according to the symbolic difference;

[0154] S508: Use the full quantity comparison difference or the incremental quantity comparison difference as the quantity comparison difference.

[0155] In this embodiment, the comparison mode parameter is the basic control factor guiding the data volume comparison operation, which defines the data range selection method and the logical path for difference generation. This mechanism supports two comparison modes, full mode and incremental mode, each corresponding to a different data selection strategy and difference calculation method. In full mode, the system uses all records in the source table and the target table as the basis for data volume comparison; in incremental mode, the system needs to parse the filter condition expression, extract the subset that meets the conditions for comparison, and focus on the data synchronization consistency within the business update window.

[0156] When the comparison mode is full, the system directly defines the full set of records in the source and target tables as the current data range. The full set of records refers to all physical records extracted from the corresponding table structure without any filtering conditions. The system counts the total number of records in the source and target tables separately. A common method includes executing an unconditional count statement, such as SELECTCOUNT(*)FROM tablename. These two values ​​serve as the total amount of source data and the total amount of target data, respectively. To eliminate interference from the direction of the difference, the absolute difference between the two values ​​is calculated. The result is the full quantity comparison difference, which quantifies the degree of deviation in data volume between the two tables.

[0157] When the comparison mode is incremental, the system first extracts a filter expression from the comparison task parameters. This expression typically limits the time range, status field, or business primary key interval of the business data. For example, a filter expression can be used to determine if "transaction_date >= current synchronization start time" or "status in('APPROVED','FAILED')". Based on the filter expression, the system then performs conditional count operations on both the source and target tables, extracting incremental record sets for the source and target tables. An incremental record set is a constrained subset of data that represents updated data within a specific time window or status.

[0158] The system counts the records in each of the two incremental record sets, generating source and target incremental counts. Unlike full-data mode, incremental comparison emphasizes the directionality and fluidity of data synchronization, and therefore preserves the sign when calculating the difference. The symbolic difference reflects the number of records in the source data that are greater or less than the target data. A positive value indicates loss in the target data if the source data has more records than the target data; a negative value indicates possible redundancy or duplicate writes in the target data if the source data has less records than the target data.

[0159] Finally, the system selects the full quantity comparison difference or the incremental quantity comparison difference as the quantity comparison difference based on the current comparison mode for use in subsequent report generation and alarm strategies.

[0160] Example: In the healthcare business, when synchronizing patient test report data from a hospital's LIS system to a regional health platform, the platform uses an incremental comparison strategy, filtering for the expression "report_date > = today's 00:00" to determine the daily data synchronization status. If the LIS contains 235 records, while the regional platform only contains 230, the system generates a count difference of +5, indicating that five records have not yet been stored, triggering an automatic re-sync task on the platform.

[0161] In the fintech sector, the credit approval system and risk control platform perform a full-volume comparison of the user credit information table, "credit_approval." The system performs a COUNT comparison every morning. Upon discovering that three records were missing from the target system, an alarm was generated. The technical team quickly identified the abnormal interruption of the data export task, preventing omissions in the credit review process. This quantitative comparison mechanism effectively ensures the continuity and integrity of the core data chain.

[0162] This embodiment introduces a data range decision mechanism based on comparison mode parameters, allowing the system to dynamically extract data volume differences under both full and incremental semantics. The full mode covers table-level volume verification and is suitable for the initial launch of the system or periodic consistency audits; the incremental mode locates small-scale synchronization deviations and is suitable for daily change synchronization and fault backtracking analysis. The difference results can serve as an auxiliary prerequisite for structure comparison and content comparison, and can also be used independently to trigger synchronization alarms and automated compensation processes. This mechanism effectively improves the efficiency, granularity, and response sensitivity of data consistency verification.

[0163] In one embodiment, the above step S60 includes:

[0164] S601: When the comparison mode parameter is the full mode, determine that the data sampling strategy is a sharding processing strategy, and divide the full record sets of the source table and the target table into multiple data shards according to the primary key range;

[0165] S602, extracting the field values ​​of all records in each data shard, and generating shard field value summary information;

[0166] S603, for each data shard, scan the corresponding target table data shard for records matching the shard field value summary information;

[0167] S604, recording the difference value of the unmatched fields for each data shard, and generating a shard content difference record;

[0168] S605: For each data shard, store the shard content difference record in a difference log library;

[0169] S606: Integrate the shard content difference records of all data shards to generate a full feature identification comparison result;

[0170] S607, when the comparison mode parameter is the incremental mode, determining that the data sampling strategy is a sampling strategy;

[0171] S608, parsing the filter condition expression in the comparison task parameter, and determining that the records in the source table that meet the filter condition expression are the source table incremental record set;

[0172] S609, determining a sampling ratio based on the data volume of the incremental record set of the source table and in combination with a preset ratio mapping table;

[0173] S610, selecting sampling records from the incremental record set of the source table according to the sampling ratio, and generating a sampling record identifier set;

[0174] S611, extracting field values ​​recorded in the sampled record identifier set and generating incremental field value summary information;

[0175] S612, scanning the target table for records matching the incremental field value summary information;

[0176] S613, recording the difference value of the unmatched field and the deviation value of the corresponding time type field, and generating an incremental feature identification comparison result;

[0177] S614: Use the full feature identifier comparison result or the incremental feature identifier comparison result as the feature identifier comparison result.

[0178] In this embodiment, the content focuses on the execution strategy of the data content comparison phase. Specifically, comparison mode parameters guide the branch selection of the sampling strategy, and sharding and sampling mechanisms are used in full and incremental scenarios, respectively, to ensure the ability to accurately compare the record content of large-scale heterogeneous tables. This process is the most data-intensive, computationally intensive, and matching-complex step in data consistency verification. Its goal is to accurately identify differences between two tables at the record level through field-level content summaries and efficient matching strategies.

[0179] When the comparison mode parameter is the full mode, the data sampling strategy is determined to be the shard processing strategy. The system constructs the logical interval of all table records based on the primary key field, and evenly divides all records in the source table and the target table into multiple data shards according to the primary key value range. When dividing, the number of shards can be dynamically determined based on the primary key value distribution, the record row number threshold, or the data page granularity. Within each data shard, the system extracts the field value set of all records in the shard from the source table, normalizes the field values, and pre-processes them to generate summary information in a unified format. The summary generation method can be based on the hash operation after field splicing, such as using SHA256 or MD5 to generate a field value summary to identify the uniqueness of a record within the shard.

[0180] A field summary match operation is performed on the corresponding data shard in the target table, and the summary value is used to find out whether a corresponding record exists in the target table. If the corresponding field value summary cannot be matched in the target shard, it means that the content of the record is missing or changed on the target side. The system records the corresponding field value and its difference details, and generates a shard content difference record. To support auditing and traceability, the difference record is written to the difference log library, which records the source shard ID, target shard ID, field name, field value, and summary comparison results. The difference records of all data shards will be aggregated and integrated to form a complete full-scale feature identification comparison result.

[0181] When the comparison mode parameter is incremental mode, the sampling strategy switches to a sampling strategy. The system first parses the predefined filter condition expression in the comparison task parameters, such as constructing logic such as "last_updated>2024-01-01AND status='VALID'" through the SQL WHERE statement, and filters out records that meet the conditions from the source table to form an incremental record set of the source table. The system determines the specific sampling ratio based on the number of records in the incremental record set and the preset ratio mapping table (for example, 20% sampling when the number of records is less than 10,000, 10% sampling between 10,000 and 100,000, etc.). The sampling ratio is a policy control parameter derived from the balance between business security and computing efficiency.

[0182] According to the sampling ratio, a number of records are randomly or stratified extracted from the incremental record set of the source table to form a sampling record identification set. The system extracts the field value content in the set and generates incremental field value summary information. Similar to the full-volume processing, this summary information is used to locate and match field values ​​in the target table. The system records the summary of field values ​​that cannot be matched in the target table, and further records the deviation value of the time type field, that is, the time difference between the corresponding time fields of the same records in the source table and the target table, such as the timestamp error exceeds the second unit. All difference information is constructed as the incremental feature identification comparison result as the basis for content consistency.

[0183] Finally, the system outputs the full feature identification comparison result or the incremental feature identification comparison result as the final feature identification comparison result according to the current comparison mode, for reference by the subsequent verification report generation module.

[0184] This embodiment achieves effective extraction and accurate comparison of large-scale table data content by dynamically selecting two strategies, sharding and sampling, under different comparison modes. The sharding processing strategy ensures full coverage and is suitable for comprehensive verification after system launch and migration; the sampling strategy implements lightweight and efficient spot checks and is suitable for incremental changes and real-time synchronization verification. Field summary generation and comparison avoids the performance bottleneck of item-by-item comparison of all field values ​​and improves the scalability of content verification. The time field deviation recording mechanism enhances the ability to detect data synchronization timeliness issues and improves the depth and dimension of data consistency assurance.

[0185] In one embodiment, the above step S10 includes:

[0186] S101, receiving a source database connection parameter set and a target database connection parameter set, wherein the connection parameter set includes a database type selection, a host address, a port number, and authentication information;

[0187] S102, performing network reachability verification on the host address in the connection parameter set of the source database to generate a source database reachability identifier;

[0188] S103, performing port open status verification on the port number in the connection parameter set of the source database to generate a source database port status identifier;

[0189] S104, performing database type adaptability verification on the authentication information in the connection parameter set of the source database, and generating a source database authentication validity identifier;

[0190] S105: When the source database reachability identifier, the source database port status identifier, and the source database authentication validity identifier are all in valid states, a source database connection test channel is established;

[0191] S106, performing network reachability verification on the host address in the connection parameter set of the target database, and generating a target database reachability identifier;

[0192] S107, performing port open status verification on the port number in the connection parameter set of the target database to generate a target database port status identifier;

[0193] S108, performing database type adaptability verification on the authentication information in the connection parameter set of the target database, and generating a target database authentication validity identifier;

[0194] S109, when the target database reachability identifier, the target database port status identifier, and the target database authentication validity identifier are all in valid states, establishing a target database connection test channel;

[0195] S110, storing the verified connection parameter set of the source database in a first configuration node of a database configuration library;

[0196] S111, storing the verified connection parameter set of the target database in the second configuration node of the database configuration library;

[0197] S112: Bind the first configuration node and the second configuration node to generate a configuration association identifier.

[0198] In this embodiment, configuring the connection parameters between the source and target databases is the starting point of the entire data consistency verification process, determining the stability of subsequent operations such as table structure extraction and data comparison. This process involves receiving connection parameters, layered validation, connection testing, and parameter persistence. Each step closely focuses on controlling connection validity and consistency in a heterogeneous database environment.

[0199] The connection parameter set is the smallest unit for structured definition of database connection behavior. It usually includes the database type (such as Oracle, MySQL, PostgreSQL, etc.), host address (IP or domain name), port number (used to access database services), and authentication information (such as user name, password, key, etc.). The system receives the source database connection parameter set and the target database connection parameter set through the configuration interface or API interface to complete the initial parameter injection. Network reachability verification of the host address is the first basic verification step. The system uses standard ICMP detection or TCP handshake test methods to check the responsiveness of the host IP or domain name, generate the source database reachability identifier and the target database reachability identifier, and determine whether the physical link is connected.

[0200] The system then performs open status verification on the port number, primarily by establishing a socket connection to determine whether the specified port is in a listening state. This prevents connection failures caused by an inactive database service or security group blocking. The verification results generate source and target database port status identifiers. Verification of the database type compatibility of the authentication information involves parsing the connection parameter format based on the database type, verifying the integrity and compliance of the username, password, certificate, or TNS descriptor. For example, Oracle databases require authentication information to include SERVICE_NAME or SID, while MySQL requires explicit binding of the HOST and USER fields.

[0201] When all three types of verification flags are valid, indicating that the database node meets the connection requirements at the network, transmission, and authentication levels, the system establishes a database connection test channel. This connection test channel can be implemented using JDBC, ODBC, DataSource, or other language-specific connection pools, executing lightweight connection verification commands (such as SELECT 1) to confirm the connection is stable.

[0202] After verification, the source database connection parameter set is written to the first configuration node in the database configuration repository, and the target database connection parameter set is written to the second configuration node. Configuration nodes are logical identifiers within the system, used to bind configurations to tasks. Finally, the system generates a configuration association identifier that binds these two configuration nodes. This identifier serves as a connection binding credential for comparing task parameters, supporting task reuse and tracking configuration consistency across multiple testing rounds.

[0203] Example: A commercial bank in the fintech sector faced data migration tasks between multiple legacy systems and a new generation of core systems. For example, the bank was migrating credit approval records from a legacy Oracle database to a unified data center based on a new PostgreSQL database architecture. To ensure consistency in the structure, number, and content of core business fields after migration, the bank implemented this method to build an automated data consistency verification platform. The system administrator first configured the source database as an Oracle instance and the target database as a PostgreSQL instance. After entering the host address, port, and authentication credentials, the system automatically checked network connectivity, port status, and authentication format. Upon successful verification, a connection channel was generated and the connection configuration was saved to the database configuration repository. The user then configured the comparison task parameters on the platform interface. In the task form, the source table "loan_approval_old" and the target table "loan_approval_new" were manually entered, and the comparison mode was selected as "Incremental Mode." The system standardized the table names and performed compatibility checks. It then connected to the source and target databases to query metadata interfaces to confirm field existence and structure. Entering the structure comparison phase, the platform retrieved the field lists, corresponding data types, and lengths from the source and target tables. Because the old system used the VARCHAR2 type and the new system used the TEXT type, the system automatically determined compatibility based on a built-in mapping table. Field length discrepancies were highlighted and integrated into a structure difference report. During the quantity comparison phase, due to the incremental mode, the system automatically parsed the filter expression in the task parameters, such as "approval_date >= '2024-01-01'," identifying 6543 matching records in the source table and 6540 in the target table. This generated a symbolic difference value of +3, indicating that the target system was missing three incremental data entries. Entering the content comparison phase, the system determined a 10% sampling ratio based on the data volume and ratio mapping table, extracting approximately 650 records from the incremental data. After generating a field value summary, a match was performed on the target table, revealing that eight records were missing from the target, three records had inconsistencies in the "approval_status" field, and five records had offsets of more than five minutes in the "approval_time" field. A detailed table of field differences was generated. Finally, the system integrates the structural differences, quantity differences and content comparison results to generate a unified verification report for archiving and tracing by the risk control and compliance audit center.

[0204] To enhance medical information sharing across multiple institutions, a regional health information platform needs to aggregate electronic health record (EHR) data from various municipal and county hospitals into a unified medical big data platform. Because hospitals utilize varying database types, including MySQL, SQL Server, and MongoDB, and differing data models, the platform must perform multi-dimensional consistency verification on the datasets uploaded by each medical institution to ensure the integrity and traceability of medical data within the central database. First, the administrator enters each hospital's database connection parameters into the system, including database type, address, port, and access credentials. The platform automatically checks whether each connection configuration is reachable, the port is open, and the authentication format matches, and then generates a connection channel. All validated connection parameters are stored in the database configuration repository, associated with the corresponding hospital's institution code, and a configuration association identifier is generated. The system then creates a data comparison task for each institution. For example, for the "patient_visit_record" table, the source table is a MySQL structure uploaded by a county-level hospital, and the target table is a standardized PostgreSQL table in the data center. The platform automatically extracted the field structures of the two tables and found that the source table field "symptom_description" was of type TEXT, while the target table field was of type VARCHAR(500). The system identified it as a field with compatible lengths but requiring a warning. When performing a quantity comparison, according to the synchronization strategy set by the platform, only new visit records added in the last 72 hours were compared, so the incremental mode was used. The filter condition expression was "visit_time>=current_timestamp-interval'72hours'". The system counted 3482 data items in the source table and 3481 in the target table, resulting in a +1 quantity difference prompt. The content comparison phase then began. Based on the data volume, the platform calculated a sampling ratio of 15%. After generating the field summary, it was discovered that one record was missing, and in two other records, the field "diagnosis_code" was spelled inconsistently in the target table. The system classified this field as a high-risk field. After the comparison was completed, a feature identification result was generated, and the integrated verification report was synchronized back to the hospital information system, where the data administrator confirmed the data repair requirements.

[0205] This embodiment ensures full-link verification of connection reachability, port availability, and authentication compliance before executing the comparison task, avoiding interruptions during the actual comparison phase due to connection failures, parameter errors, or authentication anomalies. The configuration node persistence mechanism supports multiple rounds of reuse and versioning of connection configurations, improving system operational efficiency. The configuration association identifier establishes a strong binding mechanism between the comparison task and the underlying connection configuration, achieving contextual consistency and traceability of task execution, effectively reducing configuration error rates and repeated maintenance costs.

[0206] In one embodiment, a database data verification device is provided, which corresponds one-to-one to the database data verification method in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the database data verification device of the present invention. It includes a database connection configuration module 10, a comparison task generation module 20, a table location analysis module 30, a structure difference comparison module 40, a data volume comparison module 50, a data content comparison module 60, and a verification result generation module 70. Each functional module is described in detail below:

[0207] A database connection configuration module 10 is used to configure the connection parameters of the source database and the target database;

[0208] A comparison task generation module 20 is used to set comparison task parameters and associate the comparison task parameters with the connection parameters of the source database and the connection parameters of the target database. The comparison task parameters include a source table identifier, a target table identifier, and a comparison mode parameter.

[0209] A table location and analysis module 30 is configured to determine a source table in the source database corresponding to the source table identifier, and determine a target table in the target database corresponding to the target table identifier;

[0210] A structure difference comparison module 40 is used to perform a table structure comparison operation on the source table and the target table to generate a structure difference record;

[0211] A data volume comparison module 50 is configured to determine a data range according to a comparison mode parameter, perform a data volume comparison operation on the source table and the target table based on the data range, and generate a quantity comparison difference;

[0212] A data content comparison module 60 is configured to determine a data sampling strategy according to comparison mode parameters, perform a data content comparison operation on the source table and the target table based on the sampling strategy, and generate a feature identification comparison result;

[0213] The verification result generating module 70 is used to integrate the structural difference records, quantity comparison differences and feature identification comparison results to generate a verification report.

[0214] In one embodiment, the comparison task generation module 20 is specifically configured to:

[0215] Receive the source table identifier and target table identifier input through the interactive interface, or receive the uploaded comparison template file;

[0216] When receiving the source table identifier and the target table identifier through the interactive interface, verifying the existence of the source table identifier and the target table identifier in the target database, and receiving comparison mode parameters corresponding to the verified source table identifier and the target table identifier;

[0217] When receiving the uploaded comparison template file, parsing the comparison template file to obtain the source table identifier, the target table identifier and the associated comparison mode parameters;

[0218] Generate comparison task parameters including source table identifier, target table identifier and comparison mode parameters;

[0219] The source table identifier in the comparison task parameters is associated with the connection parameters of the source database, and the target table identifier is associated with the connection parameters of the target database.

[0220] In one embodiment, the table location analysis module 30 is specifically configured to:

[0221] Parsing the naming hierarchy of the source table identifier, separating the source database name and the source table name, and generating a standardized source table identifier by case conversion and separator standardization;

[0222] Querying the metadata storage system of the source database to obtain source table structure information corresponding to the standardized source table identifier;

[0223] Parsing the naming format of the target table identifier, identifying special characters incompatible with the target database type contained in the target table identifier, replacing the special characters in the target table identifier with underscores, and generating a compatible target table identifier;

[0224] The metadata access interface of the target database is called to obtain target table structure information corresponding to the compatibility target table identifier.

[0225] In one embodiment, the structural difference comparison module 40 is specifically configured to:

[0226] Extracting a source table field name list, source table field data type definition, and source table field length constraint conditions from the source table;

[0227] Extract the target table field name list, target table field data type definition and target table field length constraint conditions from the target table;

[0228] Comparing the quantity differences between the source table field name list and the target table field name list, and generating a field quantity difference record;

[0229] Identify fields with the same names in the source table field name list and the target table field name list, and generate a set of fields with the same names;

[0230] For each field in the set of fields with the same name, query a preset database type mapping table to determine the convertibility of the source table field data type to the target table field data type, and generate a field type conversion status record;

[0231] For each field in the set of fields with the same name, verify the compatibility of the source table field length constraint condition and the target table field length constraint condition, and generate a field length matching result record;

[0232] The field quantity difference record, field type conversion status record and field length matching result record are integrated to generate the structure difference record.

[0233] In one embodiment, the data volume comparison module 50 is specifically configured to:

[0234] When the comparison mode parameter is the full mode, the data range is determined to be the full record set of the source table and the full record set of the target table;

[0235] Counting the total number of records in the full record set of the source table and the total number of records in the full record set of the target table, and generating the total amount of source data and the total amount of target data respectively;

[0236] Determine the absolute difference between the total amount of source data and the total amount of target data to generate a total amount comparison difference;

[0237] When the comparison mode parameter is the incremental mode, parsing the filter condition expression in the comparison task parameter;

[0238] Determine that the data range is a source table incremental record set consisting of records in the source table that meet the filter condition expression, and a target table incremental record set consisting of records in the target table that meet the filter condition expression;

[0239] Counting the number of records in the source table incremental record set and the number of records in the target table incremental record set, and generating a source incremental quantity and a target incremental quantity respectively;

[0240] Determining a symbolic difference between the source incremental quantity and the target incremental quantity, and generating an incremental quantity comparison difference based on the symbolic difference;

[0241] The full quantity comparison difference or the incremental quantity comparison difference is used as the quantity comparison difference.

[0242] In one embodiment, the data content comparison module 60 is specifically configured to:

[0243] When the comparison mode parameter is the full mode, the data sampling strategy is determined to be a sharding processing strategy, and the full record sets of the source table and the target table are divided into multiple data shards according to the primary key range;

[0244] Extracting the field values ​​of all records in each data shard to generate shard field value summary information;

[0245] For each data shard, scan the corresponding target table data shard for records matching the shard field value summary information;

[0246] For each data shard record, record the difference value of the unmatched fields and generate a shard content difference record;

[0247] For each data shard, the shard content difference record is stored in the difference log library;

[0248] Integrate the shard content difference records of all data shards to generate full feature identification comparison results;

[0249] When the comparison mode parameter is the incremental mode, the data sampling strategy is determined to be the sampling strategy;

[0250] Parsing the filter condition expression in the comparison task parameter, and determining the records in the source table that meet the filter condition expression as the source table incremental record set;

[0251] Determine the sampling ratio based on the data volume of the incremental record set of the source table and in combination with a preset ratio mapping table;

[0252] Selecting sampling records from the incremental record set of the source table according to the sampling ratio to generate a sampling record identifier set;

[0253] Extracting the field values ​​recorded in the sampled record identifier set to generate incremental field value summary information;

[0254] Scan the target table for records matching the incremental field value summary information;

[0255] Record the difference values ​​of unmatched fields and the deviation values ​​of corresponding time type fields to generate incremental feature identification comparison results;

[0256] The full feature identification comparison result or the incremental feature identification comparison result is used as the feature identification comparison result.

[0257] In one embodiment, the database connection configuration module 10 is specifically configured to:

[0258] Receive a source database connection parameter set and a target database connection parameter set, wherein the connection parameter set includes a database type selection, a host address, a port number, and authentication information;

[0259] Performing network reachability verification on the host address in the connection parameter set of the source database to generate a source database reachability identifier;

[0260] Performing port open status verification on the port number in the connection parameter set of the source database to generate a source database port status identifier;

[0261] Performing database type adaptability verification on the authentication information in the connection parameter set of the source database to generate a source database authentication validity identifier;

[0262] When the source database reachability identifier, the source database port status identifier, and the source database authentication validity identifier are all in valid states, establishing a source database connection test channel;

[0263] Performing network reachability verification on the host address in the connection parameter set of the target database to generate a target database reachability identifier;

[0264] Performing port open status verification on the port number in the connection parameter set of the target database to generate a target database port status identifier;

[0265] Performing database type adaptability verification on the authentication information in the connection parameter set of the target database to generate a target database authentication validity identifier;

[0266] When the target database reachability identifier, the target database port status identifier, and the target database authentication validity identifier are all in valid states, establishing a target database connection test channel;

[0267] Storing the verified connection parameter set of the source database in the first configuration node of the database configuration library;

[0268] Storing the verified connection parameter set of the target database in the second configuration node of the database configuration library;

[0269] Bind the first configuration node and the second configuration node to generate a configuration association identifier.

[0270] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, memory, network interface and database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a database data verification method.

[0271] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the user side of a database data verification method.

[0272] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0273] Configure the connection parameters of the source database and the target database;

[0274] Setting comparison task parameters and associating the comparison task parameters with connection parameters of a source database and a target database, wherein the comparison task parameters include a source table identifier, a target table identifier, and a comparison mode parameter;

[0275] Determine a source table corresponding to the source table identifier in the source database, and determine a target table corresponding to the target table identifier in the target database;

[0276] Performing a table structure comparison operation on the source table and the target table to generate a structure difference record;

[0277] Determine a data range according to a comparison mode parameter, perform a data quantity comparison operation on the source table and the target table based on the data range, and generate a quantity comparison difference;

[0278] Determine a data sampling strategy according to the comparison mode parameters, perform a data content comparison operation on the source table and the target table based on the sampling strategy, and generate a feature identification comparison result;

[0279] The structural difference records, quantity comparison differences and feature identification comparison results are integrated to generate a verification report.

[0280] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0281] Configure the connection parameters of the source database and the target database;

[0282] Setting comparison task parameters and associating the comparison task parameters with connection parameters of a source database and a target database, wherein the comparison task parameters include a source table identifier, a target table identifier, and a comparison mode parameter;

[0283] Determine a source table corresponding to the source table identifier in the source database, and determine a target table corresponding to the target table identifier in the target database;

[0284] Performing a table structure comparison operation on the source table and the target table to generate a structure difference record;

[0285] Determine a data range according to a comparison mode parameter, perform a data quantity comparison operation on the source table and the target table based on the data range, and generate a quantity comparison difference;

[0286] Determine a data sampling strategy according to the comparison mode parameters, perform a data content comparison operation on the source table and the target table based on the sampling strategy, and generate a feature identification comparison result;

[0287] The structural difference records, quantity comparison differences and feature identification comparison results are integrated to generate a verification report.

[0288] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0289] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0290] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0291] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A database data verification method, characterized in that: The following steps are involved: Configure the connection parameters of the source database and the target database; Setting comparison task parameters and associating the comparison task parameters with connection parameters of the source database and the target database, wherein the comparison task parameters include a source table identifier, a target table identifier, and a comparison mode parameter; Determine a source table corresponding to the source table identifier in the source database, and determine a target table corresponding to the target table identifier in the target database; Performing a table structure comparison operation on the source table and the target table to generate a structure difference record; Determine a data range according to a comparison mode parameter, perform a data quantity comparison operation on the source table and the target table based on the data range, and generate a quantity comparison difference; Determine a data sampling strategy according to the comparison mode parameters, perform a data content comparison operation on the source table and the target table based on the sampling strategy, and generate a feature identification comparison result; The structural difference records, quantity comparison differences and feature identification comparison results are integrated to generate a verification report.

2. The database data verification method according to claim 1, wherein: Set comparison task parameters and associate the comparison task parameters with the connection parameters of the source database and the target database. The comparison task parameters include a source table identifier, a target table identifier, and comparison mode parameters, including: Receive the source table identifier and target table identifier input through the interactive interface, or receive the uploaded comparison template file; When receiving the source table identifier and the target table identifier through the interactive interface, verifying the existence of the source table identifier and the target table identifier in the target database, and receiving comparison mode parameters corresponding to the verified source table identifier and the target table identifier; When receiving the uploaded comparison template file, parsing the comparison template file to obtain the source table identifier, the target table identifier and the associated comparison mode parameters; Generate comparison task parameters including source table identifier, target table identifier and comparison mode parameters; The source table identifier in the comparison task parameters is associated with the connection parameters of the source database, and the target table identifier is associated with the connection parameters of the target database.

3. The database data verification method according to claim 1, wherein: Determining a source table corresponding to the source table identifier in the source database and determining a target table corresponding to the target table identifier in the target database includes: Parsing the naming hierarchy of the source table identifier, separating the source database name and the source table name, and generating a standardized source table identifier by case conversion and separator standardization; Querying the metadata storage system of the source database to obtain source table structure information corresponding to the standardized source table identifier; Parsing the naming format of the target table identifier, identifying special characters incompatible with the target database type contained in the target table identifier, replacing the special characters in the target table identifier with underscores, and generating a compatible target table identifier; The metadata access interface of the target database is called to obtain target table structure information corresponding to the compatibility target table identifier.

4. The database data verification method according to claim 1, wherein: Performing a table structure comparison operation on the source table and the target table to generate a structure difference record includes: Extracting a source table field name list, source table field data type definition, and source table field length constraint conditions from the source table; Extract the target table field name list, target table field data type definition and target table field length constraint conditions from the target table; Comparing the quantity differences between the source table field name list and the target table field name list, and generating a field quantity difference record; Identify fields with the same names in the source table field name list and the target table field name list, and generate a set of fields with the same names; For each field in the set of fields with the same name, query a preset database type mapping table to determine the convertibility of the source table field data type to the target table field data type, and generate a field type conversion status record; For each field in the set of fields with the same name, verify the compatibility of the source table field length constraint condition and the target table field length constraint condition, and generate a field length matching result record; The field quantity difference record, field type conversion status record and field length matching result record are integrated to generate the structure difference record.

5. The database data verification method according to claim 1, wherein: Determining a data range according to a comparison mode parameter, performing a data quantity comparison operation on the source table and the target table based on the data range, and generating a quantity comparison difference value, including: When the comparison mode parameter is the full mode, the data range is determined to be the full record set of the source table and the full record set of the target table; Counting the total number of records in the full record set of the source table and the total number of records in the full record set of the target table, and generating the total amount of source data and the total amount of target data respectively; Determine the absolute difference between the total amount of source data and the total amount of target data to generate a total amount comparison difference; When the comparison mode parameter is the incremental mode, parsing the filter condition expression in the comparison task parameter; Determine that the data range is a source table incremental record set consisting of records in the source table that meet the filter condition expression, and a target table incremental record set consisting of records in the target table that meet the filter condition expression; Counting the number of records in the source table incremental record set and the number of records in the target table incremental record set, and generating a source incremental quantity and a target incremental quantity respectively; Determining a symbolic difference between the source incremental quantity and the target incremental quantity, and generating an incremental quantity comparison difference based on the symbolic difference; The full quantity comparison difference or the incremental quantity comparison difference is used as the quantity comparison difference.

6. The database data verification method according to claim 1, wherein: Determining a data sampling strategy according to the comparison mode parameters, performing a data content comparison operation on the source table and the target table based on the sampling strategy, and generating a feature identification comparison result, including: When the comparison mode parameter is the full mode, the data sampling strategy is determined to be a sharding processing strategy, and the full record sets of the source table and the target table are divided into multiple data shards according to the primary key range; Extracting the field values ​​of all records in each data shard to generate shard field value summary information; For each data shard, scan the corresponding target table data shard for records matching the shard field value summary information; For each data shard record, record the difference value of the unmatched fields and generate a shard content difference record; For each data shard, the shard content difference record is stored in the difference log library; Integrate the shard content difference records of all data shards to generate full feature identification comparison results; When the comparison mode parameter is the incremental mode, the data sampling strategy is determined to be the sampling strategy; Parsing the filter condition expression in the comparison task parameter, and determining the records in the source table that meet the filter condition expression as the source table incremental record set; Determine the sampling ratio based on the data volume of the incremental record set of the source table and in combination with a preset ratio mapping table; Selecting sampling records from the incremental record set of the source table according to the sampling ratio to generate a sampling record identifier set; Extracting the field values ​​recorded in the sampled record identifier set to generate incremental field value summary information; Scan the target table for records matching the incremental field value summary information; Record the difference values ​​of unmatched fields and the deviation values ​​of corresponding time type fields to generate incremental feature identification comparison results; The full feature identification comparison result or the incremental feature identification comparison result is used as the feature identification comparison result.

7. The database data verification method according to claim 1, wherein: Configure the connection parameters of the source and target databases, including: Receive a source database connection parameter set and a target database connection parameter set, wherein the connection parameter set includes a database type selection, a host address, a port number, and authentication information; Performing network reachability verification on the host address in the connection parameter set of the source database to generate a source database reachability identifier; Performing port open status verification on the port number in the connection parameter set of the source database to generate a source database port status identifier; Performing database type adaptability verification on the authentication information in the connection parameter set of the source database to generate a source database authentication validity identifier; When the source database reachability identifier, the source database port status identifier, and the source database authentication validity identifier are all in valid states, establishing a source database connection test channel; Performing network reachability verification on the host address in the connection parameter set of the target database to generate a target database reachability identifier; Performing port open status verification on the port number in the connection parameter set of the target database to generate a target database port status identifier; Performing database type adaptability verification on the authentication information in the connection parameter set of the target database to generate a target database authentication validity identifier; When the target database reachability identifier, the target database port status identifier, and the target database authentication validity identifier are all in valid states, establishing a target database connection test channel; Storing the verified connection parameter set of the source database in the first configuration node of the database configuration library; Storing the verified connection parameter set of the target database in the second configuration node of the database configuration library; Bind the first configuration node and the second configuration node to generate a configuration association identifier.

8. A database data verification device, characterized in that: The database data verification device includes: Database connection configuration module, used to configure the connection parameters of the source database and the target database; A comparison task generation module is used to set comparison task parameters and associate the comparison task parameters with the connection parameters of the source database and the connection parameters of the target database. The comparison task parameters include a source table identifier, a target table identifier, and a comparison mode parameter. A table location and analysis module is used to determine a source table in the source database corresponding to the source table identifier, and to determine a target table in the target database corresponding to the target table identifier; A structure difference comparison module is used to perform a table structure comparison operation on the source table and the target table to generate a structure difference record; A data volume comparison module is configured to determine a data range according to a comparison mode parameter, perform a data volume comparison operation on the source table and the target table based on the data range, and generate a quantity comparison difference; A data content comparison module is configured to determine a data sampling strategy according to comparison mode parameters, perform a data content comparison operation on the source table and the target table based on the sampling strategy, and generate a feature identification comparison result; The verification result generation module is used to integrate the structural difference records, quantity comparison differences and feature identification comparison results to generate a verification report.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a database data verification program stored in the memory and executable on the processor. When the database data verification program is executed by the processor, the steps of the database data verification method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a database data verification program, which, when executed by a processor, implements the steps of the database data verification method according to any one of claims 1 to 7.

Images