A method and system for automated cross-system data consistency verification and discrepancy analysis

By calculating task weight values ​​to allocate parallel execution nodes, performing data normalization, and comparing hash trees, the problems of uneven resource allocation and inaccurate difference location in cross-system data consistency verification are solved, thus improving the accuracy and efficiency of data comparison.

CN123115165APending Publication Date: 2026-11-17SHANGHAI YUANQING INFORMATION TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611403331.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-10
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing cross-system data consistency verification methods struggle to accurately distinguish between real differences and short-term synchronization differences in heterogeneous databases and environments with different business filtering standards. This results in uneven allocation of verification resources, an excessively large comparison range, and inaccurate difference location, increasing invalid alarms and the burden of manual review.

Method used

By acquiring the comparison task sets of the source system and the target system, calculating the task weight values ​​and allocating them to parallel execution nodes, generating a hash fingerprint set after data normalization, constructing a hash tree for layer-by-layer comparison, using a delayed queue to verify the suspected difference primary key set, and outputting the substantial difference data.

Benefits of technology

It has improved the uniformity, accuracy and execution efficiency of cross-system data comparison, reduced invalid data reading and writing and false alarms, and optimized the allocation of verification resources and the location of differences.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115165A_ABST
    Figure CN123115165A_ABST
Patent Text Reader

Abstract

This invention discloses an automated method and system for cross-system data consistency verification and difference analysis, relating to the field of data processing technology. It includes: acquiring comparison task sets for the source and target systems; calculating task weight values; and distributing them to parallel execution nodes according to their weights. Each node acquires the source and target datasets, performs normalization processing according to preset transformation rules to generate source and target standard datasets, and calculates row-level hash values ​​to form a first and second fingerprint set. A source and target hash tree are constructed, and when the root hash values ​​are inconsistent, node fingerprints are compared layer by layer to locate suspected differing primary key sets. The suspected differing primary key sets are stored in a delayed queue, and a re-verification comparison is performed after a preset time window. If the re-verification hash comparison results are still inconsistent, the substantially different data is output. This invention improves the efficiency of cross-system data verification and the accuracy of difference identification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for automated verification and discrepancy analysis of cross-system data consistency. Background Technology

[0002] As enterprise business systems continue to evolve, data synchronization, business distribution, and localized processing between source and target systems are becoming increasingly common. Existing cross-system data consistency verification methods often rely on scheduled tasks, manual spot checks, or full data comparisons to identify discrepancies between the two systems. These methods lack effective correlation between the verification process and the actual rate of data change, task execution time, and system synchronization latency, easily leading to uneven allocation of verification resources, excessively large comparison ranges, and inaccurate discrepancy localization. Especially in environments with heterogeneous databases, different field formats, and different business filtering criteria, traditional verification methods struggle to accurately distinguish between real differences and short-term synchronization differences while maintaining processing efficiency. This results in a lack of stable and reliable data for subsequent alarm, investigation, and remediation efforts.

[0003] Existing technologies, when performing cross-system data comparisons, often schedule verification tasks from different tables, business objects, or data sizes according to fixed cycles or fixed groupings, failing to fully utilize historical comparison times and data increment changes to measure the actual processing pressure of the tasks. When some data tables are frequently updated, have complex field content, or have long synchronization links, verification tasks tend to concentrate on consuming execution node resources, while tasks with fewer updates or lower processing pressure are not properly staggered, thus affecting overall verification efficiency. Traditional methods, after obtaining source and target data, often use direct comparison of original field values ​​or simple rule cleaning for processing. They lack a unified approach to handling differences such as null values, character spaces, field order, and business bypass data, causing non-business differences to be misjudged as data inconsistencies, increasing invalid alarms and the burden of manual review. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for automated cross-system data consistency verification and difference analysis.

[0005] In a first aspect, the present invention provides an automated method for cross-system data consistency verification and difference analysis, comprising: Obtain the comparison task set of the source system and the target system, extract the historical comparison time and data increment rate of each comparison task, calculate the task weight value of each comparison task based on the historical comparison time and data increment rate, and then allocate each comparison task to the parallel execution node according to the task weight value. Based on parallel execution nodes, the source dataset and target dataset of the corresponding comparison task are obtained. Then, the source dataset and target dataset are normalized according to the preset transformation rules to obtain the source standard dataset and target standard dataset. The hash value of each row of data in the source standard dataset and target standard dataset is calculated to generate the first fingerprint set and the second fingerprint set. Based on the first fingerprint set and the second fingerprint set, a source hash tree and a target hash tree are constructed respectively. When the first hash value of the source hash tree is inconsistent with the second hash value of the target hash tree, the node fingerprints of the source hash tree and the target hash tree are compared layer by layer downwards to extract the suspected differential primary key set where the node fingerprints do not match. Store the suspected difference primary key set in a delayed queue and record the initial timestamp. When the system time reaches the review time point determined based on the initial timestamp and the preset time window, obtain the source review data and the target review data based on the suspected difference primary key set, calculate the review hash comparison result between the source review data and the target review data, and output the substantial difference data when the review hash comparison result is inconsistent.

[0006] Secondly, the present invention provides an automated cross-system data consistency verification and difference analysis system, comprising: The task scheduling module is used to obtain the comparison task set of the source system and the target system, extract the historical comparison time and data increment rate of each comparison task, calculate the task weight value of each comparison task based on the historical comparison time and data increment rate, and then allocate each comparison task to the parallel execution node according to the task weight value. The normalization processing module is used to obtain the source dataset and target dataset of the corresponding comparison task based on the parallel execution nodes, and then normalize the source dataset and target dataset according to the preset transformation rules to obtain the source standard dataset and target standard dataset. The hash value is calculated for each row of data in the source standard dataset and target standard dataset to generate the first fingerprint set and the second fingerprint set. The difference screening module is used to construct a source hash tree and a target hash tree based on the first fingerprint set and the second fingerprint set, respectively. When the first hash value of the source hash tree is inconsistent with the second hash value of the target hash tree, the node fingerprints of the source hash tree and the target hash tree are compared layer by layer downwards to extract the suspected difference key set with mismatched node fingerprints. The anti-jitter verification module stores the suspected difference primary key set into a delayed queue and records the initial timestamp. When the system time reaches the verification time point determined based on the initial timestamp and a preset time window, it obtains the source verification data and the target verification data based on the suspected difference primary key set, calculates the verification hash comparison result between the source verification data and the target verification data, and outputs the substantial difference data when the verification hash comparison result is inconsistent.

[0007] In summary, the present invention has at least one of the following beneficial technical effects: 1. This invention obtains comparison task sets between the source and target systems, calculates task weight values ​​based on historical comparison time and data increment rate, and allocates them to parallel execution nodes accordingly, enabling task scheduling to dynamically adjust with data scale and execution load. Furthermore, it normalizes the source and target datasets using preset transformation rules to eliminate non-essential interference such as null values, formatting issues, and bypass data, and then generates row-level hash fingerprints, improving the uniformity, accuracy, and execution efficiency of cross-system data comparison.

[0008] 2. This invention constructs source and target hash trees using a first and a second fingerprint set. When the root hash value is inconsistent, it locates data blocks with abnormal node fingerprints layer by layer and extracts suspected difference primary key sets, achieving a shift from full comparison to hierarchical and precise location. By storing the suspected difference primary key sets in a delay queue and reviewing them after a preset time window, false differences caused by synchronization delays can be filtered out, outputting only the substantially different data. Furthermore, the conversion rules are optimized by combining anomaly cause labels and cold data archiving feedback. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart of an automated cross-system data consistency verification and difference analysis method provided by an embodiment of the present invention.

[0011] Figure 2 This is a schematic diagram of the two-dimensional step-by-step geometric construction of the hash tree construction and node fingerprint layer-by-layer comparison logic provided in the embodiment of the present invention.

[0012] Figure 3 This is a framework diagram of a cross-system data consistency automated verification and difference analysis system provided by an embodiment of the present invention. Detailed Implementation

[0013] The following description, in conjunction with the implementation of the present invention, is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the concept of the invention, and all such modifications and additions should fall within the protection scope of the present invention.

[0014] Example 1 This invention is applicable to scenarios where multiple heterogeneous business systems exist within an enterprise. For example, the source system uses an Oracle database to carry human resources master data, while the target system uses a MySQL database to carry localized human resources business data. Data is typically transferred between the two systems via interfaces, message queues, and scheduled synchronization tasks. Due to differences in database types, field formats, synchronization time differences, and business filtering rules, data inconsistencies are prone to occur. This implementation uses cross-system human resources data verification as an example to illustrate an automated cross-system data consistency verification and difference analysis method. This method does not simply compare each row of data from both ends in an intermediate table. Instead, it forms a continuous data processing chain around the comparison task set, source dataset, target dataset, fingerprint set, hash tree, suspected difference primary key set, and actual difference data. Through task weight allocation, normalization processing, hash tree drill-down, and delayed review, it reduces invalid data reads and writes and lowers false alarms during asynchronous synchronization.

[0015] The execution entity can be a data consistency verification service deployed in a server cluster. The server cluster includes a task scheduling module, a normalization processing module, a difference screening module, and a debouncing verification module.

[0016] This application discloses an automated method for cross-system data consistency verification and difference analysis.

[0017] Reference Figures 1 to 2 A cross-system data consistency automated verification and difference analysis method, comprising: Obtain the comparison task set of the source system and the target system, extract the historical comparison time and data increment rate of each comparison task, calculate the task weight value of each comparison task based on the historical comparison time and data increment rate, and then allocate each comparison task to the parallel execution node according to the task weight value. Specifically, both the source and target systems are pre-configured with database connection information, table mapping relationships, field mapping relationships, and preset conversion rules. Table mapping relationships can be stored in a configuration table. Each configuration record must include at least the source system table name, target system table name, source system primary key field, target system primary key field, the order of fields involved in the comparison, filter fields, query conditions, and task identifier. Taking the employee basic information table as an example, the table mapping relationships indicate the existence of field mapping relationships between the source system table PS_EMPLOYEE and the target system table HRIS_EMPLOYEE. Specifically, the source system employee identifier field EMPLID maps to the target system employee identifier field employee_id, and the source system department identifier field DEPTID maps to the target system department identifier field dept_id. After reading these configuration records, the data consistency verification service forms a set of comparison tasks. Each comparison task corresponds to a set of verification relationships between the source system tables and the target system tables.

[0018] After obtaining the comparison task sets of the source and target systems, the task scheduling module extracts the historical comparison time and data increment rate of each comparison task from the historical scheduling log. The historical scheduling log is automatically written each time a comparison task is executed, and includes the task identifier, task start time, task completion time, total amount of data read, current completion status, and execution node identifier. The historical comparison time reflects the processing time consumed by the comparison task in the existing execution environment, while the data increment rate reflects the rate of change of the corresponding data between adjacent comparison periods. These two metrics have different dimensions and cannot be directly added together. In this embodiment, a first weighting coefficient is used to convert the historical comparison time into a dimensionless first product, and a second weighting coefficient is used to convert the data increment rate into a dimensionless second product. The first weighting coefficient can be set as the reciprocal of a preset reference time. For example, if the preset reference time is ten minutes, and the historical comparison time is measured in minutes, the first product represents the proportion of the task's time relative to the reference time. The second weighting coefficient can be set as the reciprocal of a preset reference increment rate. For example, if the preset reference increment rate is 10,000 rows per minute, and the data increment rate is measured in rows per minute, the second product represents the proportion of the task's data change rate relative to the reference change rate. Both the first and second products obtained in this way are dimensionless values, and the task weight value obtained by adding them is also a dimensionless value, which can be used for load comparison between different comparison tasks.

[0019] After obtaining the task weight values ​​for each comparison task, the task scheduling module obtains the total number of available parallel execution nodes in the current cluster. Available parallel execution nodes refer to execution nodes that are online at the current scheduling time and have access to a database connection pool, memory computing space, and intermediate result write permissions. The task scheduling module constructs a corresponding number of execution queues based on the total number of nodes, with each execution queue bound to a parallel execution node. Subsequently, the task scheduling module allocates comparison tasks to execution queues according to their task weight values, ensuring that comparison tasks with larger weight values ​​do not cluster on the same parallel execution node. In actual deployment, a consistent hashing algorithm can be used to map task weight values ​​and task identifiers to hash ring intervals, and then the corresponding execution queue is determined based on the hash ring interval's location. If a new parallel execution node is added to the current cluster, only the comparison tasks within the affected hash ring interval need to be adjusted; if a parallel execution node becomes unavailable, only the comparison tasks in the execution queue corresponding to that node need to be migrated. Compared to fixed grouping, this approach allows the comparison task set to be redistributed as the total number of nodes changes, and the task weight values ​​are directly derived from historical comparison times and data increment rates, with the allocation criteria corresponding to the actual data processing pressure.

[0020] Based on parallel execution nodes, the source dataset and target dataset of the corresponding comparison task are obtained. Then, the source dataset and target dataset are normalized according to the preset transformation rules to obtain the source standard dataset and target standard dataset. The hash value of each row of data in the source standard dataset and target standard dataset is calculated to generate the first fingerprint set and the second fingerprint set. Specifically, after receiving a comparison task from the corresponding execution queue, the parallel execution node accesses the source and target systems respectively based on the source system table name, target system table name, primary key field, and query conditions in the comparison task to obtain the source and target datasets for the corresponding comparison task. The source and target datasets can be obtained in batches using a streaming query method, for example, reading 1,000 to 10,000 rows of data per batch. The cursor type can be set to read forward only to avoid excessive memory usage caused by loading the entire table at once. Each row of data in the source dataset must contain at least the source system primary key field and the source system fields used for comparison, and each row of data in the target dataset must contain at least the target system primary key field and the target system fields used for comparison. For multi-field primary keys, the normalization module does not rely on special delimiters to directly concatenate to form the business string. Instead, it writes the field length and field value in the field order during subsequent serialization processing, ensuring that the boundaries between different fields can be accurately identified by the machine.

[0021] Before the source and target datasets are compared, the normalization module performs normalization according to preset conversion rules. These rules are derived from configuration tables and built-in system rules, including rules for handling differences between heterogeneous databases and business filtering rules. For example, when the source system is an Oracle database, some null values ​​may be stored as a single space character; when the target system is a MySQL database, data with the same meaning may be represented by the database null value marker NULL. After identifying such null values, the normalization module replaces them with a unified null value identifier. Furthermore, character fields may differ in their leading and trailing spaces in the two databases. The normalization module removes leading and trailing spaces from the comparison dataset but retains valid spaces in the middle of the fields to avoid altering the actual content of fields such as names and addresses. After normalization, the source dataset is converted into a source standard dataset, and the target dataset is converted into a target standard dataset. The source and target standard datasets use the same field order, consistent null value representation, and consistent character cleaning rules to ensure comparability in subsequent hash value calculations.

[0022] The normalization module calculates hash values ​​for each row of data in both the source and target standard datasets, generating a first fingerprint set and a second fingerprint set. In one embodiment, a row of data in the source standard dataset includes the following fields: employee_id (value E10001), department_id (value D020), name (value Zhang San), and status (value A). The normalization module reads the row of data in a preset field order and forms a serialized string by adding the field length to the field content. For example, it first writes the length and value of the employee_id field, then the length and value of the department_id field, and so on, processing the name and status fields sequentially. Because the serialized string contains field boundary information, even if special characters such as asterisks, vertical bars, and commas appear in the field content, it will not cause field boundary confusion. The normalization module then applies a cryptographic hash function to hash the serialized string to obtain the row-level data fingerprint. The cryptographic hash function can be SHA-256, or other hash functions that meet the requirements of fixed output bit length and low collision probability. When using SHA-256, each row-level data fingerprint is 256 bits of binary data, usually stored as a 64-bit hexadecimal string. The row-level data fingerprints corresponding to the source standard dataset are aggregated into the first fingerprint set, and the row-level data fingerprints corresponding to the target standard dataset are aggregated into the second fingerprint set. Each row-level data fingerprint is associated with its corresponding primary key.

[0023] Based on the first fingerprint set and the second fingerprint set, a source hash tree and a target hash tree are constructed respectively. When the first hash value of the source hash tree is inconsistent with the second hash value of the target hash tree, the node fingerprints of the source hash tree and the target hash tree are compared layer by layer downwards to extract the suspected differential primary key set where the node fingerprints do not match. Specifically, after receiving the first and second fingerprint sets, the difference screening module constructs a source hash tree and a target hash tree, respectively. The construction process uses row-level data fingerprints as leaf nodes and forms multiple data blocks according to data sharding rules. Data sharding rules can be based on primary key sorting followed by sharding with a fixed number of rows, for example, forming a data block every 5,000 rows, or it can be based on primary key hash modulo sharding. To maintain comparability between the two ends, the first and second fingerprint sets use the same data sharding rules under the same comparison task. Within each data block, the row-level data fingerprints are sorted by primary key and then the block-level hash value is calculated again. The block-level hash value serves as the lower-level node in both the source and target hash trees; adjacent block-level hash values ​​are merged to calculate the hash value of the upper-level node until a top-level node is formed. The top-level node of the source hash tree is the first hash value, and the top-level node of the target hash tree is the second hash value. Since the first and second hash values ​​respectively summarize the data content under the same comparison task at both ends, when the two are consistent, the difference screening module can determine that the comparison task does not need to enter the row-by-row difference extraction under the current data snapshot; when the two are inconsistent, the node fingerprints of the source hash tree and the target hash tree are compared layer by layer.

[0024] When the first and second hash values ​​are inconsistent, the initial difference screening module first reads the next-level node fingerprints of the source and target hash trees and compares them one by one according to the data block position. Data blocks with matching node fingerprints are not drilled down further, while data blocks with mismatched node fingerprints are marked as target data blocks. For target data blocks, the initial difference screening module continues to extract the hash values ​​of their lower-level child nodes and compare them layer by layer until the record with a mismatched row-level data fingerprint is located. Records with mismatched row-level data fingerprints may exist in the source but not in the target, exist in the target but not in the source, or exist in both but with different field contents. The initial difference screening module only aggregates the primary keys corresponding to these records into a suspected difference primary key set and does not immediately output the actual difference data. Since the hash tree structure has already excluded a large number of data blocks with matching node fingerprints, the system does not need to write all consistent records to the difference intermediate table, significantly reducing the database write pressure.

[0025] Store the suspected difference primary key set into a delayed queue and record the initial timestamp. When the system time reaches the review time point determined based on the initial timestamp and the preset time window, obtain the source review data and the target review data based on the suspected difference primary key set, calculate the review hash comparison result between the source review data and the target review data, and output the substantial difference data when the review hash comparison result is inconsistent. Specifically, after the suspected differential primary key set is generated, the debouncing verification module stores the suspected differential primary key set in a delay queue and records the initial timestamp. The delay queue can be implemented using a message queue, a database delayed task table, or a memory priority queue, but its recorded content should at least include a task identifier, primary key, suspected differential status label, initial timestamp, and verification time point. The initial timestamp is the system time at the moment the suspected differential primary key is pushed into the delay queue. A preset time window is used to cover the normal synchronization delay between the source system and the target system; for example, it can be set to five minutes, ten minutes, or thirty minutes, and the specific value can be determined based on the historical delay distribution of the system synchronization link. The debouncing verification module adds the value of the preset time window to the initial timestamp to determine the verification time point. In this way, short-term inconsistencies caused by the message queue not yet being consumed, the interface task not yet being written to the database, or a brief network delay will have a chance to recover before the verification time point arrives.

[0026] When the system time reaches the review time, the anti-jitter review module triggers a point-to-point review task. This task does not rescan the entire table; instead, it performs targeted queries in both the source and target systems based on the primary key in the delay queue. The data obtained from these queries at the current time is then designated as the source and target review data, respectively. Both source and target review data undergo the same preset conversion rules as described above, and are serialized into strings using the same preset field order before calculating the review hash value. The anti-jitter review module compares the review hash value corresponding to the source and target review data to form a review hash comparison result. If the comparison results are consistent, it indicates that the initial difference is a temporary difference caused by synchronization time difference, and the corresponding record in the delay queue is marked as automatically eliminated and removed from the alert range. If the comparison results are inconsistent, the anti-jitter review module outputs the actual difference data. Substantial difference data includes task identifier, primary key, source end review data, target end review data, review hash comparison result, and review time point, which are used for subsequent display, alerts, and archiving.

[0027] Furthermore, obtain the comparison task set of the source system and the target system, and extract the historical comparison time and data increment rate of each comparison task, including: Read the table mapping relationship data between the source system and the target system, and generate a comparison task set based on the table mapping relationship data; For each comparison task in the comparison task set, retrieve the historical scheduling logs within the preset monitoring period, and parse the task start time and task completion time from the historical scheduling logs. Calculate the time span between the task completion time and the task start time, and determine the time span as the historical comparison time. Extract the total amount of comparison data from the previous period and the total amount of comparison data from the current period from the historical scheduling logs. Calculate the data difference between the total amount of comparison data in the current period and the total amount of comparison data from the previous period to obtain the data change. The execution interval between two adjacent comparison periods is analyzed, the quotient of the data change divided by the execution interval is calculated, and the quotient is determined as the data increment rate.

[0028] In one specific embodiment, the formation of the comparison task set can be implemented in a more specific way. After the data consistency verification service reads the table mapping association data between the source system and the target system, it does not directly and mechanically convert each configuration record into a task. Instead, it first verifies whether the source system table name, target system table name, primary key field, and comparison fields in the table mapping association data are complete. For configuration records with empty primary key fields, empty comparison fields, or non-existent target system table names, the task scheduling module writes them to the configuration exception log and does not include them in the comparison task set. The table mapping association data that passes the verification generates comparison tasks according to the task identifier, along with field mapping relationships and query conditions. The comparison task set generated in this way retains the flexibility of the configuration table while avoiding misalignment between the source and target datasets due to missing fields.

[0029] When extracting historical comparison time and data increment rate, the task scheduling module retrieves historical scheduling logs within a preset monitoring period. The preset monitoring period can be, for example, the last seven days, the last fourteen days, or the last thirty days; the system can set a fixed value in the configuration file based on business scale. The task start and end times in the historical scheduling logs are typically stored as millisecond-level timestamps. The task scheduling module subtracts the task start time from the task end time to obtain the time span, and determines this time span as the historical comparison time. If the same comparison task has multiple execution records within the preset monitoring period, the task scheduling module can remove failed execution records and then take the average time span of the successful execution records as the historical comparison time. This averaging process does not change the meaning of the historical comparison time; it still uses time as the unit of measurement, but it weakens the impact of occasional network jitter on the task weight value.

[0030] The extraction of the data increment rate also relies on the same historical scheduling log. The task scheduling module reads the total amount of comparison data from the previous period and the total amount of comparison data in the current period from two adjacent comparison periods. The total amount of comparison data from the previous period and the total amount of comparison data in the current period can be the number of data rows read from the source system by the same comparison task in the corresponding period, or it can be the number of data rows participating in the comparison after merging the primary keys at both ends after normalization processing. However, the same statistical caliber must be used for the same task in different periods. The task scheduling module calculates the data difference between the total amount of comparison data in the current period and the total amount of comparison data from the previous period to obtain the data change. If the data difference is negative, it means that the table corresponding to the task has been deleted or cleaned up. The system can take its absolute change as the basis for scheduling pressure; in scenarios that only focus on new pressure, negative values ​​can also be set to zero. Then, the execution interval between two adjacent comparison periods is analyzed. For example, if the previous period is executed at midnight and the next period is executed at midnight the following day, the execution interval is 1,440 minutes. The quotient of the data change divided by the execution interval is determined as the data increment rate, with the unit being rows per minute. Since the data increment rate comes from the difference in the total amount of data between two adjacent periods and the actual interval, it can reflect the speed of data change rather than simply the static size of the table.

[0031] Furthermore, based on historical comparison time and data increment rate, the task weight value of each comparison task is calculated, and then each comparison task is allocated to parallel execution nodes according to the task weight value, including: Extract the first weight coefficient of historical comparison time and the second weight coefficient of data increment rate, calculate the first product of historical comparison time and the first weight coefficient, calculate the second product of data increment rate and the second weight coefficient, and add the first product and the second product to obtain the task weight value. Get the total number of nodes available for parallel execution in the current cluster, and build a corresponding number of execution queues based on the total number of nodes; The consistent hashing algorithm is used to map task weight values ​​to hash ring intervals. Each comparison task is assigned to a corresponding execution queue according to the hash ring interval. The parallel execution nodes corresponding to the execution queues obtain the source dataset and target dataset of the corresponding comparison task.

[0032] In a specific embodiment, regarding the calculation and allocation of task weight values, this implementation uses a first weight coefficient and a second weight coefficient to normalize data of different dimensions. The first weight coefficient is multiplied by the historical comparison time to obtain a first product, and the second weight coefficient is multiplied by the data increment rate to obtain a second product. To ensure dimension matching, the first weight coefficient is taken as the reciprocal of the historical comparison time, and the second weight coefficient is taken as the reciprocal of the data increment rate. For example, if the historical comparison time is measured in minutes, the first weight coefficient is measured in minutes; if the data increment rate is measured in rows per minute, the second weight coefficient is measured in rows per minute. Thus, neither the first nor the second product carries a physical dimension, and they can be added together to obtain the task weight value. The larger the task weight value, the more computing resources the comparison task is expected to occupy. After obtaining the total number of nodes available for parallel execution in the current cluster, the task scheduling module establishes an execution queue for each parallel execution node and records the sum of the task weight values ​​currently allocated to the execution queue. When mapping task weights using the consistent hashing algorithm, the hash endpoint can be calculated by concatenating the task identifier and the task weight into a canonical string. The hash ring interval to which the hash endpoint belongs is then mapped to the execution queue. For comparison tasks with significantly higher weights, the task scheduling module can split them into several subtasks based on the primary key range and map them separately, resulting in a more balanced weight distribution among the execution queues. Comparison tasks assigned to the execution queue are read by the corresponding parallel execution nodes, and subsequent acquisition of the source and target datasets uses this allocation result as the entry point.

[0033] Furthermore, based on preset transformation rules, the source dataset and the target dataset are normalized to obtain the source standard dataset and the target standard dataset, including: Extract business attribute identifiers from the source dataset and the target dataset, and determine whether the corresponding data belongs to the bypass filter set based on the business attribute identifiers; Remove data belonging to the bypass filter set and retain the remaining comparison dataset; Identify null values ​​in the comparison dataset, replace them with a uniform null identifier, remove leading and trailing spaces from the comparison dataset, and generate the source standard dataset and the target standard dataset.

[0034] In one specific embodiment, during the normalization process, the extraction of business attribute identifiers is used to control the bypass filter set. The normalization module extracts business attribute identifiers from the source and target datasets. These identifiers can be employee IDs, department numbers, organizational attribute codes, or business line codes parsed from the primary key. In a human resources scenario, when the primary key field contains the field identifiers employee_id or EMPLID, the normalization module locates the employee ID field based on the field mapping relationship, then queries the department information of each employee in batches to obtain the department attribute code. The bypass filter set can be maintained by a configuration table. For example, data with a department attribute code of 70 represents franchise companies, while data with department attribute codes of 94, 95, 96, 97, and 98 represents international business lines. The normalization module determines whether the corresponding data belongs to the bypass filter set based on the business attribute identifiers. Data belonging to the bypass filter set is removed and not included in the comparison dataset; the remaining data is retained as the comparison dataset, and null value processing and character processing continue to be performed. Since both the source and target datasets use the same bypass filter set, data that does not need to be validated in the business will not be retained on one end and removed on the other, thus avoiding new differences introduced by inconsistent filtering standards.

[0035] The processing of null values ​​and leading / tailing spaces is completed within the comparison dataset. The normalization module scans each field in the comparison dataset one by one. When a field type allows null values ​​and the field value is the database null flag NULL, a single space, or multiple spaces, the field value is identified as null data and replaced with a unified null value identifier. The unified null value identifier can be an internally invisible control value or a fixed string, but it must be unique within the same verification task and not confused with the actual value of the business field. For character fields, the normalization module removes leading and tailing spaces from the comparison dataset; for numeric fields, it standardizes the number of decimal places; for date fields, it standardizes the time zone and format. The source and target standard datasets generated by the above processing are sent to the hash value calculation stage, with the output of the previous stage directly becoming the input of the next stage.

[0036] Furthermore, hash values ​​are calculated for each row of data in the source and target standard datasets to generate a first fingerprint set and a second fingerprint set, including: The data in each row of the source standard dataset and the target standard dataset are serialized and concatenated according to the preset field order to obtain the serialized string corresponding to each row of data. A cryptographic hash function is used to perform hash calculations on the serialized string to obtain a row-level data fingerprint; The row-level data fingerprints corresponding to the source standard dataset are aggregated into a first fingerprint set, and the row-level data fingerprints corresponding to the target standard dataset are aggregated into a second fingerprint set.

[0037] In a specific embodiment, when generating the first and second fingerprint sets, a preset field order serves to align the fields at both ends. This preset field order originates from the field mapping relationship in the comparison task and is fixed into the memory context by the normalization processing module when the task starts. For example, if the field order is primary key, department, name, and status, both the source and target standard datasets will output fields in this order. During serialization concatenation, to avoid unclear content boundaries due to special characters, the normalization processing module writes a field number, field length, and field value for each field. The field length is measured in bytes or characters, but remains consistent within the same task. After the serialized string is generated, a cryptographic hash function performs hash calculations to obtain row-level data fingerprints. The row-level data fingerprints are stored together with the primary key in the memory computing context or a lightweight intermediate table. All row-level data fingerprints corresponding to the source standard dataset are sorted according to the primary key to form the first fingerprint set, and all row-level data fingerprints corresponding to the target standard dataset are sorted according to the same sorting rule to form the second fingerprint set. Since the row-level data fingerprints do not expose the original field text, the amount of sensitive information stored in the intermediate processing stage can also be reduced.

[0038] Furthermore, a source hash tree and a target hash tree are constructed based on the first fingerprint set and the second fingerprint set, respectively. When the first hash value of the source hash tree is inconsistent with the second hash value of the target hash tree, the node fingerprints of the source hash tree and the target hash tree are compared layer by layer downwards to extract the suspected differential primary key set where the node fingerprints do not match, including: Based on the data sharding rules, the first fingerprint set and the second fingerprint set are divided into multiple data blocks. The block-level hash value of each data block is calculated. Based on the block-level hash value, a source hash tree and a target hash tree are constructed. The top-level node of the source hash tree is the first hash value, and the top-level node of the target hash tree is the second hash value. The first hash value and the second hash value are compared. If the first hash value and the second hash value do not match, the hash values ​​of the child nodes in the source hash tree and the target hash tree are extracted and compared layer by layer. Lock the target data block with inconsistent child node hash values, extract the records with mismatched travel-level data fingerprints from the target data block, and aggregate the primary keys corresponding to the records into a suspected differential primary key set.

[0039] In a specific embodiment, the hash tree construction process can be further refined according to data sharding rules. The difference screening module first divides the first fingerprint set and the second fingerprint set into multiple data blocks based on the primary key sorting results. If the data volume of a certain comparison task is small, every thousand row-level data fingerprints can be considered as one data block; if the data volume is large, every ten thousand row-level data fingerprints can be considered as one data block. The data block size is a system deployment parameter, generally set according to memory capacity and database query response time. The row-level data fingerprints within each data block are concatenated in primary key order to calculate the block-level hash value. The input of the block-level hash value only contains the row-level data fingerprints and primary key order within the data block, and does not contain the original field text. The block-level hash values ​​continue to be merged upwards to form the source hash tree and the target hash tree. The first hash value and the second hash value are located at the top-level nodes of the source hash tree and the target hash tree, respectively, representing the data digest of the entire tree.

[0040] When the first and second hash values ​​do not match, the initial difference screening module does not fall back to a full row-by-row comparison. Instead, it extracts the hash values ​​of child nodes from both the source and target hash trees and performs a layer-by-layer comparison. Each layer of comparison is based on the mismatched node in the previous layer, only accessing its child node hash values. If a child node hash value matches, the data block covered by that child node is excluded; if the child node hash values ​​do not match, the child node continues to expand downwards. The loop terminates when the row-level data fingerprint of the leaf layer is located, or when the minimum data block set by the system is located. For records with mismatched row-level data fingerprints at the leaf layer, the initial difference screening module extracts the primary key corresponding to the record and aggregates it into a suspected differential primary key set. If the minimum data block still contains multiple rows, the initial difference screening module only performs row-level data fingerprint comparison within the target data block and does not scan other data blocks that have already matched in the hash tree. This process makes the source of the suspected differential primary key set clear, and each primary key can be traced back to the mismatched node fingerprint.

[0041] Furthermore, the suspected differential primary key set is stored in a delayed queue and its initial timestamp is recorded. When the system time reaches the review time point determined based on the initial timestamp and a preset time window, the source review data and the target review data are obtained based on the suspected differential primary key set, including: Generate a suspected difference status label for each primary key in the suspected difference primary key set, push the primary key carrying the suspected difference status label into the delay queue, and extract the system time at the moment of pushing into the delay queue as the initial timestamp; The initial timestamp plus the value of the preset time window is used to determine the review time point; When the system time reaches the review time point, a point-to-point review task is triggered. Based on the primary key in the delay queue, a targeted query is performed in the source system and the target system. The data obtained from the targeted query at the current time is determined as the source review data and the target review data, respectively.

[0042] In one specific embodiment, when processing a set of suspected discrepancies in primary keys, the anti-jitter verification module generates a suspected discrepancy status label for each primary key in the set. The suspected discrepancy status label includes a task identifier, primary key, initial discovery time, and pending verification status. After a primary key carrying a suspected discrepancy status label is pushed into a delay queue, the anti-jitter verification module extracts the system time at the moment of pushing it into the delay queue as the initial timestamp. If the same primary key enters the delay queue multiple times within a preset time window, the anti-jitter verification module can merge them into a single record pending verification, retaining the earliest initial timestamp to avoid duplicate checks. The verification time point is determined by adding the value of the preset time window to the initial timestamp. The preset time window can be set according to the synchronization link from the source system to the target system; for example, a real-time message synchronization link can be set to three minutes, and a scheduled batch synchronization link can be set to thirty minutes. When the system time reaches the verification time point, the anti-jitter verification module triggers a check verification task, performing a targeted query in both the source and target systems based on the primary keys in the delay queue. The field range of the targeted query is consistent with the original comparison task. The current-time data obtained from the query is respectively identified as the source-end review data and the target-end review data. The spot-check review task only processes the suspected difference set of primary keys, so its database load is usually much lower than re-executing the full task. Furthermore, after outputting substantial difference data when the verification hash comparison results show inconsistencies, the process also includes: Extract the length of the first field in the source system and the length of the second field in the target system of the substantial difference data; When the length of the first field is greater than the length of the second field and the prefix of the content of the first field is the same as the content of the second field, generate a length truncation exception reason tag; Extract the first numerical state of the substantial difference data in the source system and the second numerical state in the target system; When the first numerical state is a numerical existence state and the second numerical state is a null value state, a data loss exception reason label is generated. The labels for reasons for length truncation and reasons for data loss are linked to the actual difference data for display.

[0043] In one specific embodiment, after outputting the substantial difference data, this implementation can also generate anomaly cause tags. The difference analysis processor reads the length of the first field in the source system and the length of the second field in the target system for the same field from the substantial difference data. The lengths of the first and second fields use the same measurement metric, such as counting by the number of characters or by the actual number of bytes in the database. If the length of the first field is greater than the length of the second field, and the prefix of the content of the first field is consistent with the content of the second field, the difference analysis processor generates a length truncation anomaly cause tag. For example, if the department name in the source system is "East China Regional Human Resources Shared Service Center" and the department name in the target system is "East China Regional Human Resources," and the field length in the target system is less than the field length in the source system, the consistent prefix can support the judgment that the field has been truncated. After this anomaly cause tag is bound to the substantial difference data, maintenance personnel can directly see the suspected field length limitation cause when viewing the substantial difference data.

[0044] The same substantial difference data can also be used for data loss classification. The difference analysis processor extracts the first numerical state of the substantial difference data in the source system and the second numerical state in the target system. The first and second numerical states are not the field values ​​themselves, but rather the existence states of the field values ​​after uniform null value marking. When the first numerical state is a numerical existence state and the second numerical state is a null value state, the difference analysis processor generates a data loss anomaly reason label. If multiple fields simultaneously meet this condition, field-level data loss anomaly reason labels can be generated separately and bound to the same substantial difference data for display. The bound display can be written to the difference record table or returned to the front-end page through a query interface; the displayed data includes at least the primary key, field name, source field value, target field value, length truncation anomaly reason label, and data loss anomaly reason label. This processing does not merely inform the two ends of the inconsistency, but rather associates the verified differences with field-level cause judgments, making subsequent repairs more evidence-based.

[0045] Furthermore, it also includes: Obtain the access frequency value of the intermediate table storing substantial difference data within a preset monitoring period; When the access frequency value is lower than the preset active threshold, the substantial difference data in the intermediate table will be marked as cold data; Extract the generation time of cold data. When the time interval between the current system time and the generation time exceeds the preset retention period, archive the cold data to obtain an archive feature set. Update the preset transformation rules of the corresponding comparison task based on the archive feature set, and clear the cold data in the intermediate table.

[0046] In one specific embodiment, for the lifecycle management of the intermediate table, this implementation continues to record the access status of the intermediate table after the substantial difference data is output. The intermediate table is used to store substantial difference data, suspected difference processing records, hash tree node summaries, or analysis results before archiving. The data cleaning processor counts the access frequency of the intermediate table within a preset monitoring period. The access frequency can be measured by the number of queries per hour or per day, and the access sources include page queries, API calls, and alarm backtracking queries. When the access frequency is lower than a preset activity threshold, the data cleaning processor marks the substantial difference data in the intermediate table as cold data. The preset activity threshold has the same unit as the access frequency; for example, if the access frequency is measured by the number of queries per day, the preset activity threshold is also measured by the number of queries per day, and there is no inconsistency in the units of measurement between the two.

[0047] After cold data is tagged, the data cleaning processor extracts the generation time of the cold data and compares it with the current system time. When the time interval between the current system time and the generation time exceeds the preset retention period, the data cleaning processor archives the cold data to obtain an archive feature set. The archive feature set is not a simple copy of the full text of the cold data, but rather extracts fields relevant to subsequent validation rule optimization, such as task identifier, difference fields, anomaly reason labels, occurrence count, first occurrence time, last occurrence time, and processing result. After the archive feature set is written to the archive database, the rule maintenance processor updates the preset transformation rules of the corresponding comparison task based on the archive feature set. For example, if a field has a substantial difference caused by leading and trailing spaces for a long time, and it has been manually confirmed that the difference has no business meaning, then a leading and trailing space cleaning rule for that field can be added to the preset transformation rules of the corresponding comparison task; if an organizational attribute has been confirmed as not requiring validation for a long time, then its business attribute identifier can be added to the bypass filter set. After completing the writing of the archive feature set and the updating of the preset transformation rules, the data cleaning processor clears the cold data in the intermediate table, thereby avoiding the infinite accumulation of cold data in the intermediate table and affecting the performance of subsequent queries and writes.

[0048] Example 2 This application discloses an automated cross-system data consistency verification and difference analysis system.

[0049] Reference Figure 3 A cross-system data consistency automated verification and difference analysis system, comprising: The task scheduling module is used to obtain the comparison task set of the source system and the target system, extract the historical comparison time and data increment rate of each comparison task, calculate the task weight value of each comparison task based on the historical comparison time and data increment rate, and then allocate each comparison task to the parallel execution node according to the task weight value. The normalization processing module is used to obtain the source dataset and target dataset of the corresponding comparison task based on the parallel execution nodes, and then normalize the source dataset and target dataset according to the preset transformation rules to obtain the source standard dataset and target standard dataset. The hash value is calculated for each row of data in the source standard dataset and target standard dataset to generate the first fingerprint set and the second fingerprint set. The difference screening module is used to construct a source hash tree and a target hash tree based on the first fingerprint set and the second fingerprint set, respectively. When the first hash value of the source hash tree is inconsistent with the second hash value of the target hash tree, the node fingerprints of the source hash tree and the target hash tree are compared layer by layer downwards to extract the suspected difference key set with mismatched node fingerprints. The anti-jitter verification module stores the suspected difference primary key set into a delayed queue and records the initial timestamp. When the system time reaches the verification time point determined based on the initial timestamp and a preset time window, it obtains the source verification data and the target verification data based on the suspected difference primary key set, calculates the verification hash comparison result between the source verification data and the target verification data, and outputs the substantial difference data when the verification hash comparison result is inconsistent.

[0050] Specifically, this embodiment also discloses a cross-system data consistency automated verification and difference analysis system. This system can be deployed on a single server or in a cluster of multiple servers. The system includes a task scheduling module, a normalization processing module, a difference initial screening module, and a debouncing verification module. Data is transmitted between these modules via internal message channels, shared database tables, or remote call interfaces. The task scheduling module connects to a configuration database and a scheduling log database to obtain comparison task sets for the source and target systems, extracts the historical comparison time and data increment rate of each comparison task, writes the calculated task weight values ​​into the scheduling context, and then allocates each comparison task to parallel execution nodes based on the task weight values. The normalization processing module is deployed on the parallel execution nodes to access the source and target systems according to the comparison tasks, obtain the source dataset and target dataset, and generate a source standard dataset and a target standard dataset according to preset transformation rules, while also generating a first fingerprint set and a second fingerprint set. The initial difference screening module receives the first and second fingerprint sets, constructs source and target hash trees, and compares node fingerprints layer by layer when the first and second hash values ​​are inconsistent, outputting a set of suspected difference primary keys. The debouncing verification module maintains a delay queue, recording the initial timestamp and verification time point. When the verification time point arrives, it acquires source and target verification data, calculates the verification hash comparison result, and outputs substantial difference data when the verification hash comparison results are inconsistent. These modules use the same task identifier and primary key as data association points during processing, maintaining a continuous and traceable data flow relationship between the comparison task set, standard dataset, fingerprint set, hash tree, suspected difference primary key set, and substantial difference data.

[0051] The above content is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described or use similar methods to replace them, as long as they do not deviate from the concept of the invention, they should all fall within the protection scope of the present invention.

[0052] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0053] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A method for automated cross-system data consistency verification and difference analysis, characterized in that, include: Obtain the comparison task set of the source system and the target system, extract the historical comparison time and data increment rate of each comparison task, calculate the task weight value of each comparison task based on the historical comparison time and data increment rate, and then allocate each comparison task to the parallel execution node according to the task weight value. Based on parallel execution nodes, the source dataset and target dataset of the corresponding comparison task are obtained. Then, the source dataset and target dataset are normalized according to the preset transformation rules to obtain the source standard dataset and target standard dataset. The hash value of each row of data in the source standard dataset and target standard dataset is calculated to generate the first fingerprint set and the second fingerprint set. Based on the first fingerprint set and the second fingerprint set, a source hash tree and a target hash tree are constructed respectively. When the first hash value of the source hash tree is inconsistent with the second hash value of the target hash tree, the node fingerprints of the source hash tree and the target hash tree are compared layer by layer downwards to extract the suspected differential primary key set where the node fingerprints do not match. Store the suspected difference primary key set in a delayed queue and record the initial timestamp. When the system time reaches the review time point determined based on the initial timestamp and the preset time window, obtain the source review data and the target review data based on the suspected difference primary key set, calculate the review hash comparison result between the source review data and the target review data, and output the substantial difference data when the review hash comparison result is inconsistent.

2. The method of claim 1, wherein, Obtain the comparison task set of the source system and the target system, and extract the historical comparison time and data increment rate of each comparison task, including: Read the table mapping relationship data between the source system and the target system, and generate a comparison task set based on the table mapping relationship data; For each comparison task in the comparison task set, retrieve the historical scheduling logs within the preset monitoring period, and parse the task start time and task completion time from the historical scheduling logs. Calculate the time span between the task completion time and the task start time, and determine the time span as the historical comparison time. Extract the total amount of comparison data from the previous period and the total amount of comparison data from the current period from the historical scheduling logs. Calculate the data difference between the total amount of comparison data in the current period and the total amount of comparison data from the previous period to obtain the data change. The execution interval between two adjacent comparison periods is analyzed, the quotient of the data change divided by the execution interval is calculated, and the quotient is determined as the data increment rate.

3. The method of claim 2, wherein, Based on historical comparison time and data increment rate, the task weight value of each comparison task is calculated, and then each comparison task is allocated to parallel execution nodes according to the task weight value, including: Extract the first weight coefficient of historical comparison time and the second weight coefficient of data increment rate, calculate the first product of historical comparison time and the first weight coefficient, calculate the second product of data increment rate and the second weight coefficient, and add the first product and the second product to obtain the task weight value. Get the total number of nodes available for parallel execution in the current cluster, and build the corresponding number of execution queues based on the total number of nodes; The consistent hashing algorithm is used to map task weight values ​​to hash ring intervals. Each comparison task is assigned to a corresponding execution queue according to the hash ring interval. The parallel execution nodes corresponding to the execution queues obtain the source dataset and target dataset of the corresponding comparison task.

4. The method of claim 1, wherein, Based on preset transformation rules, the source and target datasets are normalized to obtain the source standard dataset and the target standard dataset, including: Extract business attribute identifiers from the source dataset and the target dataset, and determine whether the corresponding data belongs to the bypass filter set based on the business attribute identifiers; Remove data belonging to the bypass filter set and retain the remaining comparison dataset; Identify null values ​​in the comparison dataset, replace them with a uniform null identifier, remove leading and trailing spaces from the comparison dataset, and generate the source standard dataset and the target standard dataset.

5. The method of claim 1, wherein, Calculate the hash value of each row of data in the source and target standard datasets to generate a first fingerprint set and a second fingerprint set, including: The data in each row of the source standard dataset and the target standard dataset are serialized and concatenated according to the preset field order to obtain the serialized string corresponding to each row of data. A cryptographic hash function is used to perform hash calculations on the serialized string to obtain a row-level data fingerprint; The row-level data fingerprints corresponding to the source standard dataset are aggregated into a first fingerprint set, and the row-level data fingerprints corresponding to the target standard dataset are aggregated into a second fingerprint set.

6. The method of claim 1, wherein, Based on the first and second fingerprint sets, a source hash tree and a target hash tree are constructed respectively. When the first hash value of the source hash tree is inconsistent with the second hash value of the target hash tree, the node fingerprints of the source and target hash trees are compared layer by layer downwards to extract the suspected differential primary key set of node fingerprint mismatches, including: Based on the data sharding rules, the first fingerprint set and the second fingerprint set are divided into multiple data blocks. The block-level hash value of each data block is calculated. Based on the block-level hash value, a source hash tree and a target hash tree are constructed. The top-level node of the source hash tree is the first hash value, and the top-level node of the target hash tree is the second hash value. The first hash value and the second hash value are compared. If the first hash value and the second hash value do not match, the hash values ​​of the child nodes in the source hash tree and the target hash tree are extracted and compared layer by layer. Lock the target data block with inconsistent child node hash values, extract the records with mismatched travel-level data fingerprints from the target data block, and aggregate the primary keys corresponding to the records into a suspected differential primary key set.

7. The method of claim 1, wherein, The suspected differential primary key set is stored in a delayed queue and its initial timestamp is recorded. When the system time reaches the review time point determined based on the initial timestamp and a preset time window, the source and target review data are obtained based on the suspected differential primary key set, including: Generate a suspected difference status label for each primary key in the suspected difference primary key set, push the primary key carrying the suspected difference status label into the delay queue, and extract the system time at the moment of pushing into the delay queue as the initial timestamp; The initial timestamp plus the value of the preset time window is used to determine the review time point; When the system time reaches the review time point, a point-to-point review task is triggered. Based on the primary key in the delay queue, a targeted query is performed in the source system and the target system. The data obtained from the targeted query at the current time is determined as the source review data and the target review data, respectively.

8. The method of claim 1, wherein, After outputting substantial difference data when the verification hash comparison results show inconsistencies, the following is also included: Extract the length of the first field in the source system and the length of the second field in the target system of the substantial difference data; When the length of the first field is greater than the length of the second field and the prefix of the content of the first field is the same as the content of the second field, generate a length truncation exception reason tag; Extract the first numerical state of the substantial difference data in the source system and the second numerical state in the target system; When the first numerical state is a numerical existence state and the second numerical state is a null value state, a data loss exception reason label is generated. The labels for reasons for length truncation and reasons for data loss are linked to the actual difference data for display.

9. The method of claim 1, wherein, Also includes: Obtain the access frequency value of the intermediate table storing substantial difference data within a preset monitoring period; When the access frequency value is lower than the preset active threshold, the substantial difference data in the intermediate table will be marked as cold data; Extract the generation time of cold data. When the time interval between the current system time and the generation time exceeds the preset retention period, archive the cold data to obtain an archive feature set. Update the preset transformation rules of the corresponding comparison task based on the archive feature set, and clear the cold data in the intermediate table.

10. The system of claim 1, wherein, A method for performing automated cross-system data consistency verification and difference analysis as described in any one of claims 1-9 includes: The task scheduling module is used to obtain the comparison task set of the source system and the target system, extract the historical comparison time and data increment rate of each comparison task, calculate the task weight value of each comparison task based on the historical comparison time and data increment rate, and then allocate each comparison task to the parallel execution node according to the task weight value. The normalization processing module is used to obtain the source dataset and target dataset of the corresponding comparison task based on the parallel execution nodes, and then normalize the source dataset and target dataset according to the preset transformation rules to obtain the source standard dataset and target standard dataset. The hash value is calculated for each row of data in the source standard dataset and target standard dataset to generate the first fingerprint set and the second fingerprint set. The difference screening module is used to construct a source hash tree and a target hash tree based on the first fingerprint set and the second fingerprint set, respectively. When the first hash value of the source hash tree is inconsistent with the second hash value of the target hash tree, the node fingerprints of the source hash tree and the target hash tree are compared layer by layer downwards to extract the suspected difference key set with mismatched node fingerprints. The anti-jitter verification module stores the suspected difference primary key set into a delayed queue and records the initial timestamp. When the system time reaches the verification time point determined based on the initial timestamp and a preset time window, it obtains the source verification data and the target verification data based on the suspected difference primary key set, calculates the verification hash comparison result between the source verification data and the target verification data, and outputs the substantial difference data when the verification hash comparison result is inconsistent.