System and method for adaptive multi-stage filtering and data consistency check with exact decoding

Abstract

The application provides a data consistency checking system and method of adaptive multi-level filtering and accurate decoding, which combines the advantages of multi-level bloom filter and IBLT, introduces an adaptive parameter adjustment mechanism based on historical statistics, and realizes fast, accurate and robust consistency checking of large-scale data sets with extremely low network overhead and computing cost.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, particularly to the field of database consistency verification, and specifically to an adaptive multi-level filtering and precise decoding data consistency verification method, system, electronic device, and computer-readable storage medium. Background Technology

[0002] In modern distributed systems, ensuring consistency among different data replicas is crucial. Traditional data consistency verification methods mainly include the following, but they all have significant shortcomings when facing the challenges of dynamically changing data scale and discrepancy rates in the real world:

[0003] 1. Full data comparison: This method involves transmitting a complete copy of the data to the other end for comparison, line by line. While accurate, this method incurs extremely high network and computational costs when dealing with massive amounts of data, making it impractical for production environments.

[0004] 2. Hash Comparison: Perform overall or block hash calculations on the data at both ends, and determine whether they match by comparing the hash values. This method can reduce network transmission, but if the hashes do not match, a large amount of data still needs to be transmitted to locate the specific difference (e.g., stepwise binary search), which is inefficient at low difference rates.

[0005] 3. Merkle Tree: Merkle trees reduce the amount of data transfer required for difference localization by constructing hash trees. However, the computational complexity of a Merkle tree is O(n log n), which still presents a performance bottleneck on large datasets. More importantly, its structure and parameters are difficult to change once determined, making it unable to adapt to dynamic changes in data size.

[0006] 4. Existing probabilistic data structure solutions: such as using a Bloom filter or an invertible Bloom lookup table (IBLT) alone:

[0007] Using a Bloom filter alone: ​​can only determine whether a difference exists, but cannot accurately locate specific differences, thus limiting its functionality;

[0008] Using IBLT alone: ​​When processing massive amounts of common elements, its internal hash collisions generate a large number of "dirty buckets" (i.e., buckets with a count of 0 but a checksum of non-zero), which can lead to decoding failures. To avoid this problem, a huge IBLT must be built that is far larger than the actual size required for the differences, thus losing its core space efficiency advantage.

[0009] Existing technologies generally employ static parameter configuration, which cannot effectively cope with dynamic changes in factors such as data scale, discrepancy rate, and network conditions in real-world scenarios. When the discrepancy rate is low, excessive configuration may waste resources; when the discrepancy rate unexpectedly increases or the data distribution is uneven, insufficient configuration may lead to verification failure. Therefore, there is an urgent need for a more intelligent, efficient, low-overhead, and adaptively adjustable data consistency verification scheme to maintain optimal performance and robustness under changing operating conditions. Summary of the Invention

[0010] To address the technical problems existing in the prior art, the present invention provides the following technical solution:

[0011] On the one hand, an adaptive multi-level filtering and precise decoding data consistency verification method is provided. This method is implemented by an electronic device and applied between at least one source and one target data copy. The method includes:

[0012] S1. Adaptive parameter generation: Based on historical verification performance metrics, a set of verification parameters is dynamically generated before each verification task begins.

[0013] S2. Multi-level filtering: Using the aforementioned verification parameters, perform at least one filtering operation on the data at the source and target ends to identify a set of suspected differential primary keys containing rows that have been added, deleted, or possibly modified.

[0014] S3. Difference Refinement: Perform precise decoding operation only on the suspected difference primary key set to parse out the final difference items between the source end and the target end;

[0015] S4. Historical Statistics: Record the performance metrics of this verification task so that subsequent verification tasks can generate adaptive parameters and form a feedback loop.

[0016] Preferably, the historical verification performance index includes at least one of the following:

[0017] Network overhead, decoding success rate, or historical discrepancy rate.

[0018] Preferably, in step S1, the step of dynamically generating a set of verification parameters based on historical verification performance indicators before the start of each verification task includes:

[0019] By using a preset composite cost function, historical verification performance indicators are analyzed and historical performance is evaluated. Based on the cost results, the verification parameters are selected from a preset strategy range and generated.

[0020] Preferably, in step S2, the multi-stage filtering includes:

[0021] First-level filtering: Identify candidate primary keys for addition and deletion by using a swapped primary key Bloom filter;

[0022] Second-level filtering: For the common primary keys that have passed the first-level filtering, a swap row content hashbloom filter is used to identify candidate primary keys whose content may have been modified.

[0023] Preferably, in step S3, before performing the precise decoding operation, the following steps are also included:

[0024] Determine whether the size of the suspected differential primary key set exceeds a preset threshold for the proportion of total data:

[0025] If the limit is exceeded, precise decoding will be stopped and the system will automatically switch to the backup full hash comparison strategy.

[0026] If the value is not exceeded, then perform precise decoding.

[0027] Preferably, the filtering operation uses a Bloom filter, a Cuckoo filter, or an XOR filter; and the precise decoding operation uses a reversible Bloom lookup table (IBLT).

[0028] Preferably, in step S3, during the precise decoding operation, an iterative adjustment step is also included:

[0029] When decoding fails, the decoding structure capacity is automatically increased and decoding is retried.

[0030] On the other hand, an adaptive multi-level filtering and precise decoding data consistency verification system is provided. This system is used to implement the aforementioned adaptive multi-level filtering and precise decoding data consistency verification method. The system includes:

[0031] The adaptive parameter generation module is used to dynamically generate a set of verification parameters based on historical verification performance metrics before the start of each verification task.

[0032] A multi-level filtering module is used to perform at least one filtering operation on the data at the source and target ends using the verification parameters to identify a set of suspected differential primary keys containing rows that have been added, deleted, or possibly modified.

[0033] The difference refinement module is used to perform precise decoding operations only on the suspected difference primary key set, and to parse out the final difference items between the source end and the target end;

[0034] The historical statistics module is used to record the performance metrics of this verification task, so that subsequent verification tasks can generate adaptive parameters and form a feedback loop.

[0035] On the other hand, a verification process is provided for the data consistency verification method or the data consistency verification system of the adaptive multi-level filtering and precise decoding described above, which is applied to multiple data partitions divided according to the primary key range or hash sharding and executed in parallel.

[0036] On the other hand, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the data consistency verification method of adaptive multi-level filtering and accurate decoding as described above is implemented.

[0037] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement the above-described adaptive multi-level filtering and accurate decoding data consistency verification method.

[0038] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0039] 1. Significantly reduced network overhead: Through an innovative multi-level filtering module, this invention can filter out over 99.9% of identical data with a very high probability, leaving only a very small number of "suspected discrepancies" for subsequent modules to process. In experiments with datasets of millions, the network transmission volume of data summaries is reduced by several orders of magnitude compared to traditional full-scale comparison schemes, effectively solving the network bottleneck problem in large-scale data verification.

[0040] 2. Superior Performance and Scalability: The core algorithm of this invention has a time complexity close to linear O(n), and key operations can be efficiently completed in memory. Verification of millions of data points can be completed in seconds, demonstrating excellent industrial-grade performance. Its modular design and natural support for data partitioning give it good horizontal scalability.

[0041] 3. Unprecedented Intelligent Adaptability and Robustness: The unique adaptive parameter control module and historical feedback closed-loop mechanism are the core advantages that distinguish this invention from existing technologies. This enables the system to automatically learn and predict future load characteristics based on historical performance, intelligently optimizing operating parameters to proactively adapt to dynamic changes in data scale and variance rate. Combined with a high variance rate switching mechanism, this invention maintains high performance and verification success rate in both typical low variance rate scenarios and extreme variance rates as high as 30%, demonstrating extremely strong robustness.

[0042] 4. High engineering practicality: This invention has a reasonable memory footprint and controllable resource consumption. Furthermore, it decouples from specific databases through a standardized data acquisition module, making it easily adaptable to various data storage systems such as SQLite, PostgreSQL, and MySQL as a general-purpose middleware. It is convenient to deploy and has a wide range of application scenarios. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of a data consistency verification method for adaptive multi-level filtering and accurate decoding provided by the present invention;

[0045] Figure 2 This is a schematic diagram of the flow mechanism of a verification method provided in an embodiment of the present invention;

[0046] Figure 3 This is a block diagram of an adaptive multi-level filtering and precise decoding data consistency verification system provided in an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0048] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0049] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0050] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0051] In this embodiment of the invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0052] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0053] The terminology is explained as follows:

[0054] Nouns / Abbreviations Explanation Bloom filters (BF) An efficient probabilistic set deduplication structure that supports fast determination of element existence, allows for a certain false positive rate, and does not support deletion. Reversible Bloom Lookup Table (IBLT) A data structure that supports set difference decoding can efficiently find set difference items when the differences are small. Primary Key (pk) A field in a database table that uniquely identifies a row. Row content hash (row_hash) The hash value calculated by concatenating all field values ​​in a row is used for content-level consistency verification. False Positive Rate (FP) The probability of a Bloom filter misclassifying a non-existent element as present. Composite cost function (Cost) A weighted function that integrates multiple metrics such as network load, decoding failure rate, and difference rate is used for adaptive parameter adjustment. Partitioning Parallel Large tables can be partitioned by primary key range or hash, and consistency checks can be performed on multiple partitions in parallel. Merkle Tree A tree-structured hash table, commonly used for data consistency verification in distributed systems. Cuckoo Filter A probabilistic set deduplication structure that supports deletion operations, with higher space efficiency than traditional Bloom filters. XOR Filter An efficient set deduplication structure based on XOR operation, supporting fast lookup and low space consumption. Rateless IBLT A dynamically scalable IBLT variant that adapts to dynamic changes in the variance rate. pg_stat_user_tables Statistics views for PostgreSQL databases can provide information such as the number of rows in a table, and the count of inserts, deletions, and modifications. information_schema.tables System tables in databases such as MySQL record table structure and statistical information. Warm Start Improve the accuracy of initial parameter prediction by utilizing existing statistical information or historical data. Cold start For the first run without historical data, parameters must be estimated based on default values ​​or statistical information.

[0055] This invention provides a data consistency verification method for adaptive multi-level filtering and accurate decoding. This method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart shown illustrates a data consistency verification method for adaptive multi-level filtering and precise decoding. The processing flow of this method may include the following steps:

[0056] S1. Adaptive parameter generation: Based on historical verification performance metrics, a set of verification parameters is dynamically generated before each verification task begins.

[0057] S2. Multi-level filtering: Using the aforementioned verification parameters, perform at least one filtering operation on the data at the source and target ends to identify a set of suspected differential primary keys containing rows that have been added, deleted, or possibly modified.

[0058] S3. Difference Refinement: Perform precise decoding operation only on the suspected difference primary key set to parse out the final difference items between the source end and the target end;

[0059] S4. Historical Statistics: Record the performance metrics of this verification task so that subsequent verification tasks can generate adaptive parameters and form a feedback loop.

[0060] This invention combines the advantages of multi-level Bloom filters and IBLT, and introduces an adaptive parameter adjustment mechanism based on historical statistics to achieve fast, accurate, and robust consistency verification of large-scale datasets with extremely low network overhead and computational cost.

[0061] The following will be combined with the appendix Figure 2 The present invention will be described in detail below with reference to a verification mechanism flow. The specific method steps and principles are as follows:

[0062] 1. Data Acquisition

[0063] It is responsible for connecting the source and target databases or data stores, and efficiently obtaining the set of data to be verified, which is usually represented as a list of tuples (primary key, row content hash).

[0064] 2. Adaptive parameter generation

[0065] One of the core innovations of this invention is that it abandons the rigid model of static parameter configuration and introduces an intelligent decision-making mechanism based on historical feedback. Before each verification task begins, it analyzes historical statistical data (such as network traffic, IBLT decoding success rate, historical difference rate, etc.) and dynamically calculates and generates the optimal combination of algorithm parameters for the current task based on a configurable composite cost function (mainly including the target false positive rate target_fp of the Bloom filter and the IBLT capacity multiplier IBLT_multiplier). This module not only passively responds to historical failures but also predicts the load and difference situation of future verification tasks based on historical trends, thereby proactively configuring optimal parameters and achieving a dynamic balance between saving resources and ensuring success rate.

[0066] 3. Multi-stage filtration

[0067] The innovative aspect of this cascading filtering strategy, which progressively eliminates the vast majority of identical data, lies in:

[0068] First-level filtering (primary key level): A primary key Bloom filter is constructed using parameters generated by the adaptive parameter control module. By swapping the primary key Bloom filter, the vast majority of "added" and "deleted" rows are quickly identified at extremely low cost.

[0069] The second-level filter (content level) constructs a row content hashbloom filter by only filtering rows that are common to both sides (i.e., the primary key exists at both ends). By swapping this filter, a small number of rows whose content may have been "modified" can be efficiently filtered out from a massive amount of rows with identical content. This two-layer filtering design, compared to single-layer filtering, can more accurately isolate suspected differences, minimizing the amount of data that ultimately needs to be processed by IBLT.

[0070] 4. Differentiated Refining

[0071] Difference Summary: Summarizes the primary keys of all "suspected differences" rows (including newly added, deleted, and potentially modified rows) filtered by the multi-level filtering module;

[0072] IBLT Precise Decoding: Using parameters generated by the adaptive parameter control module, a small IBLT containing only the aforementioned suspected differing primary keys is constructed. By differencing and decoding the two IBLTs, the final differing terms are accurately and error-free. This invention's IBLT only needs to process the highly condensed "suspected differing set" after multi-level filtering; the data volume is typically several orders of magnitude smaller than the original dataset, thus fundamentally avoiding the decoding failures or low space efficiency caused by internal collisions in traditional IBLT schemes when dealing with massive amounts of common elements.

[0073] High discrepancy rate switching mechanism: This is a key design feature to ensure the robustness of this invention. Before IBLT decoding, the system determines whether the proportion of "suspected discrepancies" to the total data exceeds a preset threshold (e.g., 30%). If it does, it indicates that extreme situations such as large-scale data synchronization or disaster recovery may have occurred, at which point the performance advantage of IBLT is no longer present. The system will automatically terminate the IBLT process and switch to the full hash comparison mode, which is more suitable for high discrepancy scenarios, thereby ensuring efficient and reliable verification under any operating conditions.

[0074] 5. Historical Statistics

[0075] As the system's "memory unit," it records key performance indicators (such as total number of rows, final difference count, network overhead, IBLT decoding success, and parameters used) in detail after each verification task and persists these records. These records provide decision-making basis for the adaptive parameter control module, forming a feedback loop for the entire system's self-optimization.

[0076] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. It should be noted that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it.

[0077] The verification method of this invention mainly relies on the collaborative work of three core logical components: DatabaseConnector, SyncStatsManager, and BloomIBLTSyncChecker. Their working principles will be described below:

[0078] (a) DatabaseConnector

[0079] This component is responsible for interacting with heterogeneous databases. In this embodiment, it is configured to connect to an SQLite database. Its core method, get_rows, is responsible for retrieving all records from the specified table and generating a unique content hash for each row.

[0080] To improve efficiency, before fully acquiring row data, the system can first perform a quick pre-check using metadata information from both databases (such as the number of rows in the table, the last update timestamp, etc.). This metadata comparison can not only efficiently discover macroscopic differences in the quantity of data between the two ends, but also provide necessary parameter basis (such as the total number of rows n) ​​for subsequent Bloom filter and IBLT algorithm modules, thereby achieving more accurate and efficient data consistency verification.

[0081] The process of generating the row content hash is as follows:

[0082] 1. Query `PRAGMA table_info(table_name)` to get all column names and their standard order in the database;

[0083] 2. Execute `SELECT FROM table_name ORDER BY pk_column` to ensure that the data on both ends has a consistent traversal order;

[0084] 3. In the database or application, for each row of data, concatenate all column values ​​(pk_value, col1_value, col2_value...) strictly according to the column order retrieved from the database, using a specific and unique internal delimiter (such as |) to form a long string. During this process, NULL values ​​are deterministically converted to a predefined special string (such as "__NULL__") to avoid ambiguity;

[0085] 4. Hash the string using a high-speed, low-collision hash algorithm (such as SHA256) to generate a fixed-length line content hash digest.

[0086] Ultimately, this component outputs a tuple of (primary key, row content hash) for each row, which can be used by subsequent modules.

[0087] (ii) SyncStatsManager (Statistics and Adaptive Manager)

[0088] This component is the brain behind the "intelligent adaptive" feature of this invention, and its working mechanism is as follows:

[0089] 1. History Record: It maintains a JSON file (e.g., sync_stats.json) to persistently store statistics for the most recent N (e.g., 10) validation tasks. Each record is a snapshot of the key results after a validation task is completed, and contains at least the following fields:

[0090] total_rows: The total number of data rows actually involved in the comparison during this verification task;

[0091] diff_count: The final number of difference data points detected in this verification;

[0092] IBLT_success: A boolean flag indicating whether the IBLT decoding was successful;

[0093] network_bytes: The total number of bytes transferred over the network during this verification;

[0094] params_used: The core parameters used in this verification, such as {'target_fp':1e-4,'IBLT_multiplier':1.5}.

[0095] 2. Composite Cost Function: At the start of each verification task, the `get_adaptive_params` method loads historical data and uses a composite cost function to quantitatively assess the system's recent "health status":

[0096] Cost=w1 NetworkBurden+w2 IBLTFailureRate+w3 OverDetectionRate

[0097] w1, w2, and w3 are configurable weights. For example, setting them to 0.5, 0.4, and 0.1 respectively indicates a high focus on network overhead and decoding failures. These weights can be adjusted by the system administrator according to business needs (e.g., network cost-sensitive vs. absolute success rate-first).

[0098] NetworkBurden: The normalized network burden calculated based on historical network_bytes / total_rows. For example, it is obtained by dividing the average network byte per row over the most recent N times by an ideal baseline value (such as 8 bytes / row).

[0099] IBLTFailureRate: The percentage of historical instances where IBLT_success was false. This is a very strong penalty.

[0100] OverDetectionRate: The average of historical diff_count / total_rows, reflecting the “concentration” or “entropy” of the differential data that needs to be processed by IBLT.

[0101] 3. Parameter Generation: Based on the calculated Cost value, the system dynamically generates the parameters for this task, referencing a set of preset policy ranges. This is not a simple linear mapping, but a rule-based decision-making process.

[0102] Cold start (no historical data): Uses a set of conservative and general default parameters, such as target_fp=1e-5,IBLT_multiplier=2.0;

[0103] Cost > 0.7 (High Cost / Dangerous Range): Indicates that the system has been performing poorly recently.

[0104] If IBLTFailureRate > 0, it is determined that decoding failure is the main cause. The system enters the "ensure success at all costs" mode: greatly tighten target_fp to 1e-6 (reduce BF false positives, negative for IBLT), and at the same time increase IBLT_multiplier to 3.0 (provide a large amount of redundancy).

[0105] If IBLTFailureRate == 0, it is determined that excessive network overhead is the main cause. The system enters the "optimize network" mode: appropriately tighten target_fp to 1e-5, and at the same time keep IBLT_multiplier or slightly increase it to 2.0.

[0106] 0.3 < Cost ≤ 0.7 (medium cost / observation interval): It is determined that the system performance is stable but there is room for optimization. Adopt standard parameters, such as setting target_fp to 1e-4 and IBLT_multiplier to 1.8.

[0107] Cost ≤ 0.3 (low cost / high efficiency interval): It is determined that the system has performed excellently recently. It can enter the "save resources" mode and appropriately relax the parameters to reduce computing and network overhead, for example, setting target_fp to 1e-3 and IBLT_multiplier to 1.5.

[0108] Through this closed-loop, the system can continuously learn from the actual operation effects, dynamically adjust its own configuration, and achieve long-term, efficient, and stable data consistency verification.

[0109] (III) BloomIBLTSyncChecker (core checker)

[0110] This component is responsible for executing the complete consistency verification process, as shown in the appendix Figure 2 as follows.

[0111] 1. Bloom filter construction: It calls a mature Bloom filter library (such as pybloom_live) or uses SQL implementation to create a Bloom filter according to the target_fp and the number of data rows n provided by SyncStatsManager. The number of bits m required for the filter and the number of hash functions k are automatically calculated by the library according to the publicly available optimal formulas without manual intervention.

[0112] m = - (n ln p) / (ln 2) 2 (where p is target_fp);

[0113] k = ceil((m / n) ln2).

[0114] 2. IBLT Construction: It uses a pure Python implementation of the IBLT class. The number of buckets is determined by `buckets = max(10, int(len(suspect_pks))`. The IBLT_multiplier is dynamically determined to ensure a minimum capacity even in cases where there are very few suspected differences. The size of the key and value is fixed based on the digest length of the hash algorithm (e.g., 32 bytes for SHA256).

[0115] 3. Network Traffic Calculation: After the process is completed, it accurately calculates the network overhead of this task for historical statistics. The calculation method is as follows:

[0116] BF network overhead = (BF.num_bits / 8);

[0117] IBLT network overhead = buckets (4-byte counter + 32-byte key hash + 32-byte value hash);

[0118] Total cost = BF_pk_A + BF_pk_B + BF_hash_A + BF_hash_B + IBLT_A + IBLT_B.

[0119] To maximize network efficiency, the basic exchange process described above can be optimized. For example, an efficient asymmetric process can be adopted: First, source A sends its primary key Bloom filter to target B (first transmission); B compares the filters, identifies the candidates for addition / deletion, constructs its content hash Bloom filter based on the primary key shared by both parties, and then sends it back to A (second transmission). Under this optimized process, only two Bloom filters need to be transmitted to complete the two-stage filtering, halving the network overhead. The calculations in the embodiments will be based on this optimized process.

[0120] The theoretical basis for the high difference rate switching mechanism of this invention is as follows:

[0121] The "automatic switching for high difference rates" mechanism in this invention is a key engineering decision based on a deep understanding of the performance of IBLT and full hash comparison technologies in different scenarios. This is not simply a backup plan, but a core design element ensuring the system remains efficient and robust under all operating conditions. Its theoretical basis is as follows:

[0122] 1. Algorithm complexity and applicable scenarios:

[0123] IBLT's core operations (insertion and decoding) have a time and space complexity of O(s), where s is the number of suspected discrepancies to be processed. Therefore, IBLT has an unparalleled efficiency advantage in scenarios with low discrepancy rates (i.e., s is much smaller than the total data volume n).

[0124] Full hash comparison: This method requires building complete hash tables (such as dictionaries or HashMaps) for both ends of the data, and its time and space complexity are both O(n). Regardless of the difference, its cost is proportional to the total amount of data n.

[0125] 2. Performance Crossover Point Analysis:

[0126] When the difference rate is extremely low (s≪n), IBLT only needs to process a very small amount of data, and its resource consumption is far lower than that of a full hash comparison that requires processing all n data.

[0127] As the difference rate increases, s gradually approaches n. At this point, the complexity advantage of IBLT diminishes, and its performance begins to approach that of full hash comparison.

[0128] When the difference rate reaches a high threshold (e.g., 30% or higher), the performance of IBLT (which requires processing close to n data points) is comparable to that of full hash comparison. Considering that full hash comparison is simpler and more direct to implement, lacking the complex internal structure and decoding overhead of IBLT, switching to full hash comparison may actually be a more efficient and reliable choice in engineering practice. This threshold (e.g., 30%) is an empirical value based on theoretical estimation and multiple experimental comparisons.

[0129] In summary, by setting a reasonable switching threshold, this invention can intelligently identify extreme scenarios where IBLT no longer offers performance advantages and automatically downgrade to a full hash alignment scheme more suitable for handling high-concentration differential data. This ensures that, as a whole, this invention can automatically select the optimal path regardless of the data distribution and differential situation, thereby achieving comprehensive performance optimization and robustness.

[0130] Different embodiments are provided below to implement this solution.

[0131] Example 1: Standard and efficient verification process under large scale and low difference rate

[0132] This embodiment simulates a typical and valuable industrial scenario: verifying two databases with millions of records each, where the difference rate is extremely low.

[0133] Data conditions:

[0134] Source A: 1,000,000 records;

[0135] Target B: 1,000,001 records. Compared to A, B has 333 deleted records, 334 newly added records, and 333 records modified. The total difference involves 1,000 primary keys. The difference rate is approximately 0.1%.

[0136] Historical conditions: Assuming the system's historical records are stable, the composite cost calculated by SyncStatsManager is 0.35 (medium cost).

[0137] Verification process:

[0138] 1. Parameter generation: The system enters the "moderate adjustment" strategy and generates the following parameters: target_fp=1e-4,IBLT_multiplier=1.8.

[0139] 2. First-level filtering (primary key - BF):

[0140] A and B each create a primary key Bloom filter with a capacity of 1,000,000 and a false positive rate of 1e-4. The network size of each BF is approximately 2.3MB.

[0141] After swapping BF, terminal A accurately identified 333 "deleted" primary keys, while terminal B accurately identified 334 "new" primary keys.

[0142] 3. Second-level filtering (row hash - BF):

[0143] Based on 1,000,000 - 333 = 999,667 "non-deleted" rows, the content hash BF is constructed on side A, with a size of approximately 2.3MB.

[0144] The B-end uses its "non-new" rows for querying. Since target_fp=1e-4, in addition to the 333 actually modified rows, it may also identify (999667) additional rows. 1e-4)≈100 rows with "pseudo-modifications" due to hash collisions. Therefore, a total of approximately 333+100=433 primary keys with "potential modifications" were identified.

[0145] 4. Differential Refining (IBLT):

[0146] Summary of suspected discrepancies: The size of the suspect_pks set is approximately 333(del) + 334(add) + 433(mod) ≈ 1100.

[0147] Check the switching threshold: 1100 / 1,000,000=0.11%, which is much less than 30%, so do not switch.

[0148] Constructing IBLT: number of buckets = max(10, int(1100)) 1.8))=1980. This is a relatively small IBLT, with a network size of approximately 1980. (4+32+32) bytes ≈ 135KB.

[0149] Decoding: IBLT_A - IBLT_B decoding successful. IBLT can perfectly handle about 100 common elements with "pseudo-modifications" (which cancel each other out in the two IBLTs) and output the exact A_only set (333 deleted, 333 original values).

[0150] And the B_only set (334 new values ​​and 333 modified values).

[0151] 5. Record statistics: Store the total_rows=1,000,000, diff_count=1000, IBLT_success=true, and the calculated total network overhead (based on the optimized asymmetric process, a total of 2 BF entities were transmitted, with an overhead of approximately 4.6MB; plus IBLTs overhead of approximately 0.27MB, for a total of approximately 4.9MB) into sync_stats.json.

[0152] Conclusion: This embodiment clearly demonstrates that when processing massive amounts of data (millions of records), the present invention has a performance advantage of several orders of magnitude over the prior art.

[0153] Example 2: Adaptive adjustment triggered by decoding failure

[0154] This embodiment simulates a more challenging scenario: 500,000 records, and the previous verification failed due to hash collisions and other reasons, causing IBLT decoding to fail.

[0155] Data conditions: There are approximately 500,000 records each at source A and target B.

[0156] Historical condition: The most recent record in sync_stats.json has IBLT_success set to false.

[0157] Verification process:

[0158] 1. Parameter generation:

[0159] SyncStatsManager read historical records showing IBLT_success=false, causing an increase in the IBLT FailureRate. The calculated composite cost spiked above 0.7, entering the "high-cost / danger zone" strategy.

[0160] The system generates stricter parameters: target_fp=1e-6, IBLT_multiplier=3.0. 2. First and second level filtering:

[0161] The system uses a false positive rate of 1e-6 to construct a Bloom filter. Compared to 1e-4, the individual data size of the Bloom filter is larger but the overall size is smaller. However, it can filter more accurately, significantly reducing the number of false positive primary keys entering the IBLT, thereby reducing the decoding difficulty of the IBLT.

[0162] 3. Differential Refining (IBLT):

[0163] Assuming that after filtering, the size of the suspect_pks set is 550 (500 real differences + 50 false differences caused by BF false positives).

[0164] Constructing IBLT: number of buckets = max(10, int(550)) 3.0))=1650. The upgrade of IBLT_multiplier from the usual 1.8 to 3.0 means that IBLT is allocated far more redundant space than usual, greatly enhancing its collision resistance and decoding capabilities. Due to the reduction in the number of suspected differences, the size of this IBLT (approximately 0.21MB) is also smaller than that of Example 1.

[0165] Decoding: Thanks to more precise filtering (reduced noise) and a larger capacity IBLT (enhanced robustness), the decoding was successful.

[0166] 4. Record Statistics: Record the IBLT_success=true value, along with the corresponding parameters and network overhead for this round. In this task, the network overhead per data row increased due to the stricter false alarm rate and the larger IBLT multiplier, but the total network overhead (approximately 3.4MB for BFs + approximately 0.21MB for IBLT, totaling approximately 3.6MB) was still lower than in Example 1 because the total data volume was halved.

[0167] Conclusion: This embodiment demonstrates the effectiveness of the feedback loop and adaptive mechanism of the present invention. The system can "learn" from failures and automatically adopt more robust parameters (lower p, larger IBLT_multiplier) to ensure the success rate of subsequent verification tasks. Although in this example, the total resource overhead decreased due to the halving of the total data volume, this parameter adjustment strategy reflects that the system is prepared to increase resource input per unit of data in exchange for system stability when necessary.

[0168] Example 3: Adaptive switching in high difference rate scenarios

[0169] This embodiment simulates scenarios such as disaster recovery or initial synchronization, in which the data at both ends differs greatly.

[0170] Data conditions: Source A: 10,000 records. Target B: 7,000 records. The discrepancy rate is over 30%.

[0171] Historical conditions: irrelevant.

[0172] Verification process:

[0173] 1. Parameter generation and filtering: Generate parameters according to the standard process and perform two-level filtering.

[0174] 2. Check the switching threshold:

[0175] The total number of suspect_pks easily exceeded 10,000. 30% = 3000.

[0176] The judgment logic within the core validator BloomIBLTSyncChecker is triggered if len(suspect_pks) / total_rows>0.3.

[0177] The system prints logs such as "Difference rate is too high, automatically switch to full hash comparison mode, skip IBLT".

[0178] 3. Implement the backup plan:

[0179] Instead of constructing and decoding the IBLT, the system executes a more traditional, alternative full hash comparison process: A and B each construct a dictionary (Hash Map) in memory for all rows' (primary key, row content hash), and then compare them to find differences. Although this method has higher network overhead, its overall overhead and complexity are better than a huge IBLT that is on the verge of decoding failure when the difference rate is extremely high.

[0180] Conclusion: This embodiment demonstrates the engineering practicality and fail-safe mechanism of the present invention. By setting a switching threshold, the system can intelligently identify extreme scenarios unsuitable for IBLT and automatically downgrade to a more suitable traditional solution, ensuring robustness and efficiency under any circumstances.

[0181] Example 4: Cold Start Optimization Deeply Coupled with Database Statistics

[0182] This example demonstrates how the system can perform an intelligent "warm start" when there are no historical verification records (i.e., sync_stats.json is empty or does not exist).

[0183] Data conditions: arbitrary.

[0184] Historical conditions: No historical verification records.

[0185] Verification process:

[0186] 1. Parameter generation:

[0187] SyncStatsManager detects no historical records and triggers cold start logic.

[0188] It will call a special method of DatabaseConnector that connects to the database and queries its internal statistics view (such as pg_stat_user_tables for PostgreSQL or information_schema.tables for MySQL).

[0189] From these views, one can obtain a relatively accurate total number of rows n, as well as the approximate number of tuples added, deleted, and modified since the last analysis (n_tup_ins, n_tup_del, n_tup_hot_upd).

[0190] Based on this statistical information, the system estimates the initial difference rate, for example:

[0191] estimated_diff_rate=(n_tup_ins+n_tup_del+n_tup_hot_upd) / n.

[0192] Based on this estimated difference rate, the system selects a set of initial parameters that are more targeted than the general default values. For example, if the estimated difference rate is high, target_fp=1e-5,IBLT_multiplier=2.0 are used directly.

[0193] 2. Subsequent process: Continue with the filtering and decoding process.

[0194] Conclusion: This embodiment demonstrates the intelligent optimization of the present invention in a cold start scenario. By coupling with the underlying statistical information of the database, the system avoids "blind guessing" during the first run, instead executing based on a more reasonable initial parameter set supported by data, thereby improving the success rate and efficiency of the first verification.

[0195] In addition to the above embodiments, this application may also include the following implementation methods and equivalent variations. To further broaden the scope of protection of this invention, those skilled in the art should understand that, without departing from the core idea of ​​this invention, the following equivalent substitutions or improvements are possible, and all such variations should fall within the scope of protection of this invention:

[0196] 1. Partitioned Parallel Processing: For extremely large data tables, the data can be pre-divided into multiple logical partitions based on the primary key range (e.g., pk < 1,000,000) or primary key hash partitioning (e.g., hash(pk) mod 16). The complete verification process described in this invention can be executed independently and in parallel on each partition. This significantly reduces the amount of data processed by a single verification task, thereby reducing the risk of a single IBLT decoding failure and greatly improving the overall verification speed.

[0197] 2. Iterative Adjustment of IBLT Capacity: As a supplement or alternative to the historical feedback adaptive mechanism, the IBLT capacity can be adjusted using an iterative strategy. That is, initially, a smaller IBLT_multiplier is used to construct the IBLT. If decoding fails, this multiplier is gradually increased (e.g., from 1.5 to 2.0, then to 2.5) and decoding is retried until successful. This "progressive expansion" strategy can dynamically find the minimum resource overhead required for decoding in a single task.

[0198] 3. Phased IBLT Transmission Strategy: As a network optimization variant of "iterative tuning," the system can generate two IBLTs of different capacities locally at once: a small IBLT based on optimistic estimation (using a smaller IBLT_multiplier) and a large IBLT based on pessimistic estimation (using a larger IBLT_multiplier). During verification, only the small IBLT is exchanged first. The receiver attempts to decode using the small IBLT. If decoding is successful, the verification is completed with very low network overhead; if decoding fails, the receiver requests the sender to transmit the pre-generated large IBLT for decoding. This strategy benefits from the network advantages of the small IBLT in common low-discrepancy-rate scenarios, and in the rare case of decoding failure, it quickly recovers with an additional network request, avoiding the overhead of recalculating the IBLT.

[0199] 4. Row Content Hash Generation Method: The "row content hash" mentioned in this invention can be generated in various ways. Besides concatenating all column values ​​into a string in a predetermined order before hashing, a column-level Merkle tree can also be used. That is, the hash of each column value is calculated separately, and these column hashes are then combined into a Merkle tree, with the root hash used as the final row content hash. This method may support finer-grained difference location in the future (e.g., directly locating the difference column).

[0200] 5. Variations of the Filtering and Decoding Process: This invention describes the core process of "Primary Key BF -> Content Hash BF -> IBLT". One equivalent variation could be:

[0201] Step 1: Use only the primary key Bloom filter to filter out the set of primary keys that are common to both parties;

[0202] Step 2: For this set of shared primary keys, construct a content IBLT, where the Key is the primary key and the Value is the hash of the row content. By decoding this IBLT, the primary key whose content has been modified can be directly identified.

[0203] This variant process also follows the core idea of ​​"layer-by-layer filtering and gradual refinement".

[0204] Therefore, although Bloom filters (BF) and reversible Bloom lookup tables (IBLTs) are disclosed as data structures in the prior art, the inventiveness of this invention does not stem from a simple accumulation of these data structures, but from their organic combination into a robust, system-level solution with dynamic adaptability. The fundamental reason why this invention can achieve orders-of-magnitude performance improvements and possesses high engineering feasibility lies in the synergistic effect of one or more of the following key innovations:

[0205] 1. Innovative System Architecture: An Adaptive Feedback Closed-Loop Verification System. This invention is the first to propose and implement a complete feedback closed-loop system comprising "historical statistics -> cost evaluation -> dynamic parameter tuning -> execution verification -> record statistics". This architecture transforms the data verification process from a static, fixed workflow into an intelligent process capable of self-learning and self-optimization, thereby fundamentally solving the pain point of existing technologies being unable to adapt to dynamic operating conditions.

[0206] 2. Innovative Filtering Strategy: A Two-Layer Cascaded Filter. This invention creatively designs a two-layer filtering strategy consisting of a "primary key Bloom filter + content hash Bloom filter." This strategy performs two consecutive filters on different dimensions (existence and content), achieving higher accuracy and a lower false positive rate compared to existing single-layer filtering techniques. This minimizes the "noise" in the data requiring precise decoding, creating a prerequisite for efficient and successful IBLT decoding.

[0207] 3. Innovation in Engineering Robustness: An Automatic Switching Mechanism Based on Performance Crossover Points. This invention deeply understands the performance differences of different algorithms under various data distributions and designs an automatic switching mechanism for high dissimilarity rates based on this understanding. This mechanism can determine in real-time whether IBLT is in its optimal operating range. When a high dissimilarity rate scenario is detected, the system automatically switches to the more robust full hash comparison. This design ensures that the system can select the optimal execution path under all foreseeable operating conditions, possessing engineering robustness not found in existing technical solutions.

[0208] Figure 3This is a block diagram of an adaptive multi-level filtering and precise decoding data consistency verification system according to an exemplary embodiment. The system is used for an adaptive multi-level filtering and precise decoding data consistency verification method. (Refer to...) Figure 3 The system includes an adaptive parameter generation module 310, a multi-level filtering module 320, a difference refinement module 330, and a historical statistics module 340. Among them:

[0209] The adaptive parameter generation module is used to dynamically generate a set of verification parameters based on historical verification performance metrics before the start of each verification task.

[0210] A multi-level filtering module is used to perform at least one filtering operation on the data at the source and target ends using the verification parameters to identify a set of suspected differential primary keys containing rows that have been added, deleted, or possibly modified.

[0211] The difference refinement module is used to perform precise decoding operations only on the suspected difference primary key set, and to parse out the final difference items between the source end and the target end;

[0212] The historical statistics module is used to record the performance metrics of this verification task, so that subsequent verification tasks can generate adaptive parameters and form a feedback loop.

[0213] For the working principles and interactions of the above modules, please refer to the corresponding steps in the above description; they will not be repeated here.

[0214] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 4 As shown, the electronic device may include the above-mentioned Figure 3 The illustrated adaptive multi-level filtering and precise decoding data consistency verification system. Optionally, the electronic device 410 may include a first processor 2001.

[0215] Optionally, the electronic device 410 may also include a memory 2002 and a transceiver 2003.

[0216] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0217] The following is combined with Figure 4 A detailed description of each component of electronic device 410 is provided below:

[0218] The first processor 2001 is the control center of the electronic device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0219] Optionally, the first processor 2001 can perform various functions of the electronic device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0220] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0221] In a specific implementation, as one example, the electronic device 410 may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0222] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0223] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently and be connected via the interface circuit of the electronic device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0224] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0225] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0226] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected via the interface circuit of the electronic device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0227] It should be noted that, Figure 4 The structure of the electronic device 410 shown does not constitute a limitation on the router. Actual knowledge structure identification devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0228] Furthermore, the technical effect of the electronic device 410 can be referred to the technical effect of the data consistency verification method of adaptive multi-level filtering and accurate decoding described in the above method embodiments, and will not be repeated here.

[0229] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0230] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0231] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0232] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0233] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0234] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0235] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0236] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, systems, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0237] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0238] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0239] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0240] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0241] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for data consistency check with adaptive multi-stage filtering and exact decoding, applied between at least one source and one target of data replica, characterized in that, The method comprises: S1, adaptive parameter generation: based on historical verification performance indicators, dynamically generating a set of verification parameters before each verification task starts; S2, multi-level filtering: using the verification parameters, performing at least one filtering operation on the data of the source end and the target end, comprising: First-level filtering: identifying candidate primary keys that are added and deleted through an exchange primary key Bloom filter; Second-level filtering: identifying candidate primary keys whose contents may be modified through an exchange row content hash Bloom filter on the common primary keys that pass the first-level filtering; S3, difference refining: only for the suspected difference primary key set containing added, deleted, and possibly modified rows, performing an accurate decoding operation to analyze the final difference items between the source end and the target end; S4, historical statistics: recording the performance indicators of this verification task to generate adaptive parameters for subsequent verification tasks, forming a feedback loop.

2. The method of adaptive multi-stage filtering and exact-decoding data coherency checking of claim 1, wherein, The historical verification performance indicators include at least one of the following indicators: network overhead, decoding success rate, or historical difference rate.

3. The method of adaptive multi-stage filtering and exact-decoding data coherency checking of claim 1, wherein, In step S1, the adaptive parameter generation based on historical verification performance indicators before each verification task starts comprises: analyzing historical verification performance indicators and evaluating historical performance through a preset composite cost function, and selecting and generating the verification parameters from a preset strategy interval according to the calculated composite cost function value.

4. The method of adaptive multi-stage filtering and exact-decoding data coherency checking of claim 1, wherein, Before the accurate decoding operation in step S3, it further comprises: judging whether the size of the suspected difference primary key set exceeds a preset data total amount proportion threshold: if it exceeds, aborting the accurate decoding and automatically switching to a backup full amount hash comparison strategy; if it does not exceed, performing the accurate decoding operation.

5. The method of adaptive multi-stage filtering and exact-decoding data coherency checking of claim 4, wherein, The accurate decoding operation uses a reversible Bloom lookup table.

6. The method of adaptive multi-stage filtering and exact-decoding data coherency checking of claim 1, wherein, In step S3, during the accurate decoding operation, an iterative adjustment step is further included: when decoding fails, automatically increasing the decoding structure capacity and retrying decoding.

7. The method of adaptive multi-stage filtering and exact-decoding data coherency checking of any of claims 1-6, wherein, The method is applied to multiple data partitions divided according to primary key ranges or hash shards and is executed in parallel.

8. A system for adaptive multi-level filtering and exact-decoding data consistency checking, the system for adaptive multi-level filtering and exact-decoding data consistency checking being configured to implement the method for adaptive multi-level filtering and exact-decoding data consistency checking according to any one of claims 1 to 6, characterized in that, The system comprises: an adaptive parameter generation module for dynamically generating a set of verification parameters before each verification task starts based on historical verification performance indicators; a multi-level filtering module for using the verification parameters to perform at least one filtering operation on the data of the source end and the target end to identify a suspected difference primary key set containing added, deleted, and possibly modified rows; a difference refining module for only performing an accurate decoding operation on the suspected difference primary key set to analyze the final difference items between the source end and the target end; a historical statistics module for recording the performance indicators of this verification task to generate adaptive parameters for subsequent verification tasks, forming a feedback loop.

9. An electronic device, comprising: The electronic device comprises: a processor; a memory having computer readable instructions stored thereon, which, when executed by the processor, implement the method of any one of claims 1 to 6.

Images