A spatially coupled adaptive multi-view disparity positioning system and method

Abstract

The application provides a kind of adaptive multi-view difference positioning system and method based on space coupling, the application is determined by the continuous window mapping (space coupling), the fault signal of data partition is restricted in local window, the global pollution problem of traditional IBLT random hash is solved when facing local severe failure;Second, a plurality of independent observation views are constructed, the intersection operation of multi-view hot area is used, the exponential compression of candidate partition is realized (from (W·r) times compression to (W·r) V times in single view) Finally, based on the energy distribution form index, the adaptive gating is carried out, the centralized or dispersed mode of the fault is automatically identified, and the optimal repair strategy is selected. The application significantly reduces the metadata overhead, and improves the accuracy, efficiency and robustness of local fault positioning in large-scale distributed systems.

Description

Technical Field

[0001] This invention relates to the field of view processing technology, and in particular to a data difference detection method based on spatial coupling, a fault partition identification method based on multi-view intersection, an adaptive gating and data repair decision method, a data synchronization system device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] The following is an introduction to data discrepancy detection and fault location in existing large-scale distributed systems: I. Existing Technological Foundation and Application Scenarios Reversible Bloom Lookup Tables (IBLTs), as a probabilistic data structure, are widely used in data synchronization and reconciliation scenarios due to their ability to identify discrepancies in sets with low network overhead. Their core advantage lies in significantly reducing bandwidth consumption compared to full data transmission. In addition, traditional methods such as Markle trees are also commonly used for data discrepancy detection, but they all have certain limitations.

[0003] II. Core Issues of Existing Technologies 1. Global contamination problem of traditional IBLT Traditional IBLT uses a random hash function to evenly distribute data items across the global table structure. In the case of a "local explosion" failure in a distributed system (i.e., a few partitions produce a large number of differences due to failure, while other partitions are normal), the massive differences of a single failed partition will randomly spread to all locations in the table, causing most buckets in the table to be "contaminated". The difference signals of normal partitions are masked, and the system's detection capability is completely ineffective.

[0004] 2. Redundancy problem of candidate set in single-view localization When the total number of partitions (P) is much larger than the number of segments in the difference detection table (L), even with the spatial coupling mechanism (Scheme 1), a single view will still produce a "Halo Effect" due to hash collisions, resulting in an excessively large candidate partition size (approximately P×(W·r), where W is the window width and r is the hot zone ratio), making it difficult to accurately locate faulty partitions.

[0005] 3. Insufficient fault mode adaptability Traditional methods fail to consider the diversity of difference distribution patterns: they lack adaptive decision-making mechanisms in scenarios of "centralized failures" (differences are concentrated in a few partitions) and "diffuse failures" (differences are evenly distributed across a large number of partitions). Forcing precise localization in diffuse noise scenarios leads to ineffective computation, reducing system efficiency and robustness.

[0006] III. Shift in Technological Goals and Innovation Motivation The core objective of traditional IBLT-based technology is to "fully recover all differential keys," but this can easily lead to table overflow and decoding failure when the differential scale is huge. Summary of the Invention

[0007] To address the technical problems existing in the prior art, the present invention provides the following technical solution: This invention provides an adaptive multi-view difference localization system and method based on spatial coupling for large-scale distributed systems. The invention employs deterministic continuous window mapping, multi-view intersection localization, and adaptive morphological gating to achieve isolation and precise localization of local faults. Compared to traditional data difference detection and repair methods (such as standard IBLT and Markle-tree), this invention significantly improves localization accuracy and system robustness in scenarios with severe local faults.

[0008] The overall technical framework of this invention comprises a complete "detection-location-decision" combined method through the following three progressive technical features: On the one hand, a data difference detection method based on spatial coupling is provided: [See "Solution 1: Data Difference Detection Based on Spatial Coupling" for details] 1. Spatially Coupled Window Mapping: This invention first discloses a method for deterministically mapping data partitions to a continuous, non-rebound window in the difference detection table. This method strictly constrains the fault energy of each partition to a local level, fundamentally solving the problem of global contamination that easily occurs when traditional random hash IBLT faces severe local faults.

[0009] On the other hand, a fault partitioning identification method based on multi-view intersection is provided: [See "Solution 2: A Fast Fault Partition Identification Based on Multi-View Intersection"] 2. Multi-View Intersection Localization: Based on spatial coupling, this invention further discloses a fast fault partition identification method based on multi-view intersection. By back-mapping candidate partitions to the hotspot areas of each independent view and finding the intersection of the candidate sets of multiple views, an exponential compression of the candidate partition set at the (W·r)V level is achieved, thereby significantly improving the accuracy of localization.

[0010] On the other hand, an adaptive data discrepancy repair decision-making method is also provided: [See “Solution 3: Adaptive Gating and Scene Adaptation Based on Energy Distribution Characteristics”] 3. Energy-Topological Gating Decision: Finally, to enable the system to intelligently adapt to different fault modes, this invention also discloses an adaptive gating method based on energy distribution patterns (such as the Keff / L index). It can quickly predict whether the current difference belongs to a “centralized” or “diffuse” scenario before high-cost positioning operations are executed, and automatically select the optimal subsequent strategy (perform precise positioning or degrade to global repair), thereby avoiding invalid calculations under diffuse noise and significantly improving the robustness and efficiency of the system.

[0011] These three technical features together constitute a complete technical system from basic detection and precise positioning to intelligent decision-making.

[0012] On the other hand, a data synchronization system is also provided, including: Probe construction module: used to execute the spatial coupling-based data difference detection method described above at the source and target ends, construct difference detection tables respectively, and optionally execute the multi-view construction of the fault partition identification method based on multi-view intersection described above; Difference and Recognition Module: Used to receive the difference detection table, perform difference operations, and execute the fault partition recognition method based on multi-view intersection described above to identify the fault partition candidate set; Repair Decision Module: Used to perform corresponding subsequent data repair operations based on the candidate set of fault partitions, or based on the decision output by the adaptive data difference repair decision method described above, including performing precise checksum, version rollback and / or full snapshot synchronization on the candidate partitions.

[0013] On the other hand, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method described above.

[0014] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the above method.

[0015] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: This invention solves the global contamination problem of traditional IBLT random hashing when facing severe local faults by constraining the fault signals of data partitions to local windows through deterministic continuous window mapping (spatial coupling). Secondly, it constructs multiple independent observation views and uses the intersection operation of hotspot regions of multiple views to achieve exponential compression of candidate partitions (from (W·r) times of a single view to (W·r). V Finally, based on energy distribution morphology indicators, adaptive gating is performed to automatically identify centralized or diffuse fault patterns and select the optimal repair strategy. This invention significantly reduces metadata overhead and improves the accuracy, efficiency, and robustness of local fault location in large-scale distributed systems. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a diagram illustrating the natural partitioning of data in an existing large-scale distributed storage system; Figure 2 This is a schematic diagram of the spatial coupling mapping principle of SC-IBLT in this application (showing the core mechanism of Scheme 1, namely how to deterministically map a data partition to a continuous local window, and contrasting it with the random hashing of traditional IBLT, highlighting the innovation of the "spatial coupling" of this invention). Figure 3 This is a schematic diagram illustrating the principle of multi-view intersection compression in this invention (used to demonstrate the core idea of ​​Scheme 2. It intuitively explains why the multi-view mechanism can effectively filter false positives (noise) introduced by hash collisions, because the real faulty partitions will generate strong signals in all views, while the noisy partitions only occasionally become hotspots in a single view and are eventually screened out during the intersection process, thereby achieving exponential compression of the candidate set). Figure 4This is a comparison diagram of energy distribution patterns provided by an embodiment of the present invention (displayed as a violin plot). Using only the energy distribution indices of the probe (Keff / L and Gini), it is possible to clearly distinguish between two difference modes: "bimodal" and "diffuse," even without knowing the true difference distribution. This is the experimental basis for the success of Scheme 3 (adaptive gating), proving that the system possesses a reliable classification index that requires no prior knowledge. In contrast, other statistical indices such as TOP-K (the top K indices, such as energy fraction), Gini coefficient, and entropy (such as normalized entropy) are not sufficiently discriminative. Bimodal is a bimodal distribution, and diffuse is a diffuse distribution. Figure 5 This invention provides an effectiveness verification graph for a gating mechanism (using a scatter plot to correlate "energy morphology index" (horizontal axis: Keff / L) with "localization performance" (vertical axis: ratio of correctly identified partitions to candidate set size). It demonstrates the value of the gating strategy: in regions with low Keff / L values ​​(red dots), localization gains are significant; while in regions with high Keff / L values ​​(blue dots), localization gains are negligible. This provides experimental evidence for setting the gating threshold (τlo, τhi). Figure 6 This invention provides a performance ROC curve for a gated classifier (quantitatively evaluating the performance of an adaptive gated classifier. The AUC (area under the curve) value of 0.998 demonstrates the effectiveness of the classifier, indicating a very low probability of misclassification). Detailed Implementation

[0018] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0019] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0020] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0021] In this embodiment of the invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. Unless the difference is emphasized, the meaning they express is the same. The symbol “^” represents an exponent.

[0022] To make the technical problems, technical solutions and advantages of the present invention clearer, the application and implementation of the three solutions of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Option 1: Data Difference Detection Based on Spatial Coupling Technical Background and Problem Statement An Invertible Bloom Lookup Table (IBLT) is a probabilistic data structure widely used in scenarios such as efficient set reconciliation and data synchronization. It can find the differences between two sets with lower network overhead, offering a significant bandwidth advantage compared to full data transmission. Traditional IBLTs use a random hash function to evenly distribute data items across the global table structure; this design performs well in scenarios with equal and uniformly distributed differences.

[0024] However, in existing large-scale distributed storage systems, such as Figure 1 As shown, data is naturally divided into multiple partitions, and failures are usually localized—that is, a failure in a few partitions or nodes causes a large amount of data within that partition to differ, while other partitions remain normal. In this situation, the random hashing mechanism of standard IBLT will have serious problems: the massive differences from a single failed partition will be randomly distributed to all locations in the table, causing most of the buckets in the table to be "contaminated," making it impossible to effectively identify differences in other normal partitions, thus causing the entire system's detection capability to fail.

[0025] This invention discloses a data difference detection method based on spatial coupling, aiming to overcome the aforementioned problems. The key innovation of this invention lies in the redefinition of the problem objective. The core objective of traditional IBLT-based set coordination techniques is to fully recover all difference keys, which can lead to table overflow and decoding failure when the difference scale is large. This invention proposes a fundamentally different solution: abandoning the precise recovery of all difference keys in the first stage, instead prioritizing the efficient location of "fault partitions" that generate large-scale differences. This shift in objective enables this invention to handle difference scenarios far exceeding the capacity of traditional IBLT, and is particularly suitable for the "localized explosion" failure mode commonly seen in real-world distributed systems.

[0026] This method employs a deterministic continuous window mapping mechanism (spatial coupling design) to strictly confine each partition to a local continuous region within the table, achieving fault isolation and localization. The spatial coupling design originates from related concepts in coding theory, characterized by confining all keys of the same partition to a local window, thus concentrating the energy of differences. When a partition has a large number of differences, the energy value of its corresponding window will be relatively high, while the energy of other regions remains at a lower level. This makes the signal generated by the actual differences more separable from background noise. Furthermore, compared to traditional random IBLT, this method can maintain the integrity of the rest of the table when facing localized severe faults, and due to the locality of access patterns, it improves the CPU cache hit rate during data insertion and updates.

[0027] Specific implementation steps The specific implementation process of this invention includes the following main stages (such as...) Figure 2 (As shown): 1.1 First, logical data partitioning is performed. All data items in the source and target datasets to be compared are logically divided into P independent partitions based on their logical unit identifiers (i.e., partition IDs). In distributed systems, partitions are the basic units for data management and disaster recovery, so this partitioning naturally aligns with most application scenarios. For simplicity, the mapping relationship between partitions and keys in this paper can be determined by the simplified formula pid(key) = key / / partitionstride, where partitionstride is a preset parameter used to control the size of the key-value space for each partition; pid: the partition identifier (Partition ID), representing the logical partition number to which the data item belongs. For example, if partitionstride = 1000, all data items with key values ​​in the range [0, 999] will be assigned to the same partition, those with key values ​​in the range [1000, 1999] will be assigned to the next partition, and so on. After calculation using this formula, each data item is assigned to a unique partition, which is the core identifier for data partition management; / / : represents integer division; key: refers to the key of the data item, which is the index value used to uniquely identify the data item in the dataset. In actual distributed systems, the key is usually globally unique, such as the primary key in a database or the object identifier l, pid(key) in distributed storage. There are usually deterministic rules between the key and the key, such as in a hash table.

[0028] 1.2 Next, the system executes the step of constructing the difference detection table. A difference detection table called a Spatially Coupled Invertible Bloom Lookup Table (SC-IBLT) is constructed for each dataset. Logically, this table is a one-dimensional array consisting of L consecutive "sections". Each section s (where s∈[0, L-1]) contains m0 cells. Therefore, the total number of cells in the table is M = L × m0.

[0029] Each cell contains a core field: `count` (required field): Records the number of keys inserted into this cell. In a two-way difference scenario, if a key exists in the source set A but not in the target set B, its contribution to `count` is +1; otherwise, it is -1. The initial value of `count` is 0. The `count` field is a required field for difference detection, used to determine which segments contain differences and the degree of concentration of differences.

[0030] Each cell contains two optional fields: .keysum (checksum of the key, optional field): This is typically the sum of the XOR values ​​of all keys inserted into this cell. The keysum is crucial data that IBLT can use to recover key values ​​and is required during the data recovery phase.

[0031] `.valuesum` (value checksum, optional field): Used to record the cumulative sum of values ​​associated with a key. `value(key)` represents the numerical representation of the value associated with the key or its hash value. The specific accumulation method depends on the application scenario and can be arithmetic sum, XOR sum, or other reversible operations. `valuesum` is required during the data recovery phase.

[0032] Field use case description: The `count` field is a required field for fault partition difference detection, used in the fault localization phase; `keysum` and `valuesum` are optional fields, only needed during data recovery after differences are detected. This design allows `keysum` and `valuesum` to be omitted in lightweight observation probes to reduce overhead, while retaining these fields in the repair phase where data recovery is required. In actual deployments, these fields can be implemented with different integer types, such as `count` using `int16` or `int32`, and `keysum` and `valuesum` using `int64` or `uint64`, to balance accuracy and storage overhead.

[0033] 1.3 Subsequently, this invention performs deterministic and local mapping from partitions to segments. For any partition, it is associated with a window consisting of W consecutive segments in the difference detection table through a deterministic mapping function. The specific mapping rule is as follows: calculate the window base segment index: base = H(pid) mod (L - W + 1), where H(pid) is one or more preset deterministic hash functions that map from the partition ID space to non-negative integers, L is the total number of segments in the difference detection table, and W is the window width (coupling width), representing the number of consecutive segments affected by the partition, usually W∈ [1, 8]. The modulus (L - W + 1) is used instead of L to ensure that the end position of the window [base, base+W-1] never exceeds L-1. Once base is determined, the difference information of all key-value pairs within the partition will be strictly limited to this consecutive window within the segment index range [base, base+W-1]. Any key from this partition will not be mapped to a segment outside the window. This continuous window mapping mechanism contrasts sharply with the standard IBLT, which randomly hashes keys to any location in the entire table. Its core advantages are: achieving fault isolation (a fault or anomaly in a single partition will only cause "energy concentration" within its corresponding window, without polluting other areas of the table), improving cache friendliness (accesses to all keys in the same partition are concentrated in contiguous memory addresses, significantly improving CPU L1 / L2 cache hit rate), and making the signal clear (the impact of a large fault partition will form sharp energy peaks locally, which are easy to identify from background noise).

[0034] Based on this, the system performs a key-to-cell hash mapping. For each key to be encoded within a partition pid, k independent hash functions (e.g., h1, h2, ..., h3) are used. k The key is mapped to k distinct cells within a contiguous window covered by its partition. Specifically, for the j-th hash function (j ∈ [1, k]), the offset relative to the window is calculated. ,at this time Falling on [0, W× Within the range. Then calculate the absolute cell position in the difference detection table: , where base is the window base segment index corresponding to the partition pid.

[0035] 1.4 For each defined cell [cellidx], perform the standard IBLT operation: if it is an "insertion" operation (i.e., the key comes from A but is not in B), then perform an increment operation: , , , The definitions are as follows: cell[cellidx] Meaning: A cell in the SC-IBLT (Signature-Independent Detection Table) used to store difference information.

[0036] count field Meaning: Counter, used to record the number of keys inserted into the unit.

[0037] Operation logic: If the key exists in A but not in B, count += 1; If the key exists in B but not in A, count -= 1 (decreasing operation); The initial value is 0.

[0038] keysum field Meaning: Key checksum, which is calculated by summing the keys of all inserted units using an XOR operation; Operation logic: keysum ^= key (equivalent to keysum = keysum XOR key), used to retrieve the key value in the subsequent data recovery stage.

[0039] valuesum field Meaning: Value checksum, the cumulative sum of values ​​associated with the key; Operation logic: valuesum += value(key), where value(key) is the numerical representation or hash value of the value corresponding to the key. The accumulation method (such as arithmetic sum, XOR sum) depends on the application scenario.

[0040] key and value(key) key: A unique identifier for a data item (such as a database primary key), globally unique; value(key): The value associated with the key, the specific form of which is defined by the business scenario (such as a number, string hash, etc.); Operation scenario: The above formula only applies to the "insert" operation (a key that exists in A but not in B).

[0041] Field functions: count is used for fault location, keysum and valuesum are used for data recovery, and the three together constitute the core storage structure of SC-IBLT.

[0042] 1.5 If the operation is a "delete" operation (i.e., the key comes from B but is not in A), then the opposite decrement operation is performed (count -= 1, keysum ^= key, valuesum -= value(key)). This process ensures that all keys within the same partition are mapped to a certain position in its window, while the influence of keys from a specific partition is strictly limited to its own mapping window.

[0043] Finally, the system generates a difference statistics table. On the two systems to be compared (e.g., system A and system B), separate SC-IBLT tables, denoted as SCIBLTA and SCIBLTB, are independently constructed using the exact same parameters (P, L, m0, W, k, and all hash functions). One of these tables is transmitted over the network, and at the receiving end, the corresponding units are subtracted one by one to obtain a difference table. SCIBLTdiff.cell[i] = SCIBLTA.cell[i] - SCIBLTB.cell[i], where i is the table number.

[0044] The core objective of the first approach of this invention is to utilize this difference table for fault localization, rather than recovering all differences. This difference table is primarily used for subsequent statistical analysis, such as calculating the energy score of each segment (as described in approach 2), thereby identifying "hotspot regions" carrying a large number of differences. As an optional verification step, pure cells (cells with a count field of +1 or -1) in the difference table can be decoded to extract a small number of difference samples for rapid diagnosis. However, the preferred embodiment of this approach does not require the recovery of all difference keys in the first stage; the complete difference recovery task is clearly delineated for processing in subsequent repair stages.

[0045] Summary of Technological Innovation In summary, the core technological innovations of this solution are reflected in several aspects. First, it adopts a deterministic continuous window mapping mechanism. By strictly constraining each partition to within W consecutive segments and using the formula base = H(pid)mod (L-W+1), it achieves complete localization of the partition's influence, which is fundamentally different from the random hashing of traditional IBLT. Second, by using (L-W+1) as the modulus, it designs a non-rebounding boundary processing, simplifying implementation and avoiding the additional overhead of circular arrays. Third, through a three-layer design of count (multiplicity), keysum (XOR check), and valuesum (value check), it supports standard IBLT decoding and provides sufficient statistical information for subsequent energy estimation and fault location.

[0046] Option 2: Rapid Fault Partition Identification Based on Multi-View Intersection Technical Background and Problem Statement Building upon Scheme 1, this invention further discloses a fast fault partition identification method based on the intersection of multiple views. This method aims to address the problem of insufficient localization caused by excessive candidate partitions generated by a single SC-IBLT (Scheme 1) due to hash collisions (i.e., the "halo effect") when the total number of partitions P is much larger than the number of segments L. This method achieves exponential compression of the candidate set size by constructing multiple independent, lightweight observation views and finding the intersection of hotspot results from each view.

[0047] Specific implementation steps (such as) Figure 3 (As shown) Step 1: Construct Multi-View Probes To achieve multi-angle observation of faulty partitions, the system constructs V independent observation probes in parallel (e.g., V=2 or 3). Each probe is an SC-IBLT structure, but its function focuses on fault signal collection and fault partition localization, rather than data recovery. The count field is a necessary field for difference detection, used to determine which segments contain differences and the degree of concentration of differences; while the keysum and valuesum fields are used to recover the actual data from the detected differences and are only needed in the subsequent data recovery stage. Therefore, in a preferred embodiment, these observation probes can adopt a lightweight SketchSC structure, whose cells only contain the count field, omitting the keysum and valuesum fields. This allows for complete fault localization while significantly reducing the additional memory overhead caused by multiple views.

[0048] Specifically, the memory consumption of each observation probe is reduced from "3 fields × L × m0" to "1 field × L × m0" (a reduction of approximately 87.5%) without affecting the accuracy of fault partition location. This is of great significance for network transmission efficiency and memory usage.

[0049] Step 2: Achieve view independence To ensure the orthogonality of the information provided by each view, the hash mappings of partition IDs for each view must be independent. This is achieved by equipping each view j (j∈ [0, V-1]) with an independent hash function Hj. For the same partition ID pid, the base segment basej it maps to is different in each view j: basej = Hj(pid) mod (L - W + 1), Hj can be instantiated using different hash seeds (salt / seed) to ensure that the hash results of Hj are unrelated to those of Hi (when i≠j).

[0050] In a specific implementation, the independence of multiple views can be simulated by applying an affine shuffle function (Pid shuffledj) to the partition ID. For view j, the definition is: Pid shuffled j = (aj * pid + bj) mod P, The choice of aj and bj is crucial to ensuring the bijective property of the mapping. aj should be coprime with the total number of partitions P, thus guaranteeing a one-to-one correspondence between affine transformations modulo P. When P is a power of 2 (common in practical applications), aj only needs to be odd to satisfy the coprime requirement. bj is a random or pseudo-random offset value used to disrupt the order of the original partition IDs. Different views j correspond to different (aj, bj) pairs, and these parameters are generated by the view's seed.

[0051] Substitute pidshuffledj into the segment index calculation in step three: basej = H(pidshuffledj) mod (L - W + 1), In this way, the same partition will be mapped to different segment window positions in different views, thus producing different count matrices.

[0052] Step 3: Calculate the segment energy score (CountScore) for each view. After obtaining the count value matrix for each view probe through differential calculus, an energy score Es needs to be calculated for each segment s of each view to quantify the "differential signal strength" carried by that segment. To eliminate background noise (i.e., "drift") caused by imbalances in both datasets where only A-only (A-only, present on A side but not on B-only, present on B side but not on A-only), this invention employs a second-order energy formula (CountScore) to calculate the score.

[0053] The specific steps are as follows: 1. Aggregate information within a segment: For each segment s in view j, first aggregate the count values ​​of its m0 internal cells to obtain the statistics for that segment: (Jinghe) (sum of squares), in, This represents the value of the `count` field in the `i`th cell of the `s`th segment in the difference table, specifically the key in the source set A mapped to that cell. 2. Calculate the drift-free energy score: The energy score `Es` for this segment is calculated as follows: , Physical meaning: This formula essentially calculates the variance of the count values ​​within a segment (multiplied by the degrees of freedom). After removing the influence of the mean (i.e., "drift"), it more clearly reflects whether there is a true concentration of differences within the segment.

[0054] Robustness explanation: In the "clean" segment, which has no real differences and only random hash collisions, the distribution of count values ​​is close to random, and its expected value of Es is close to 0. In the segment mapped by the "dirty" partition containing a large number of differences, the count values ​​will show large concentrated changes, resulting in a relatively higher Es.

[0055] Why is drift correction necessary? If a simple solution is used, such as simply using... (Jinghe) or The sum of squares (SQS), as an energy fraction, can cause problems in certain variance patterns. When the dataset only contains additions without deletions (A-only scenario) or only deletions without additions (B-only scenario), the count values ​​of all segments will systematically shift. For example, in a pure A-only scenario, the count value of the entire table increases by a positive constant, causing the SQS of all segments to shift. All segments are raised to similar levels, making it difficult to distinguish truly "dirty" segments using absolute values. This invention employs a drift-removal CountScore formula. By subtracting the contribution of mean shift, local discrepancy clusters can still be identified even in the presence of systematic shifts, thus improving the adaptability to various discrepancy patterns.

[0056] Step 4: Filter hotspot areas and find their intersection (e.g., Figure 3 (As shown) 1. Generate a view-level candidate set: For each view j, sort all its segments in descending order based on their energy scores Es, and select the top t segments with the highest energy as the "hot sections" of that view: , Where r is the preset hot zone ratio (e.g., r=1 / 64, 1 / 32, etc.), is a system parameter.

[0057] 2. Reverse Mapping to Partitions: A reverse mapping table is used to transform the set of hotspot segments for each view j into a set Cj of candidate partition IDs. The reverse mapping table is constructed as follows: For each view j and each partition pid: basej = Hj (pid) mod (L -W + 1) For offset ∈ [0, W-1]: sectiontopids [j] [ basej+ offset] .append( pid). The reverse mapping answers the question, "If a segment is identified as a hotspot, which partition IDs might be mapped to that segment?" This mapping is one-to-many because windows of multiple partitions may cover the same segment.

[0058] It should be noted that the theoretical bound of the space complexity of this reverse mapping table is O(P·V·W), where P is the total number of partitions, V is the number of views, and W is the window width. In practical applications, since V and W are usually small constants (e.g., 2⁻⁴), its space overhead can be considered to be linearly related to the total number of partitions P. This table structure belongs to the system's control plane data and can usually be pre-computed and generated offline or during system startup. In scenarios with extremely limited memory, there is also an alternative implementation of "computational reverse mapping," which does not store the table but instead identifies candidate partitions by traversing all partition IDs and calculating their window positions after identifying hot segments. This is a typical strategy of trading computation for space.

[0059] The candidate set Cj is defined as follows: .

[0060] 3. Intersection Calculation (Core Compression Step): The final fault partition candidate set C is the intersection of the candidate sets Cj of all V views: C = C0 ∩ C 1 ∩ . . . ∩ C {V-1}. Compression Effect Analysis and the Necessity of Multiple Views: This method exponentially compresses the size of the candidate set. Theoretical analysis shows that in the case of a single view, due to hash collisions (Halo), the size of the candidate set is approximately P×(r×W). After adding a second view, the number of partitions that are determined to be "hot" by both views is approximately P×(r×W). 2 This achieves second-order compression. Adding a third view further enhances the compression to P×(r×W). 3In practical applications, taking (V=2, W=2, r=1 / 64) as an example, the size of the candidate set for a single view is approximately P×(W×r) ≈ P×3.1%, while the intersection of two views can compress the candidate set score to the order of 0.1%, achieving an effective compression of tens of times.

[0061] Limitations of Single-View Approach: While Scheme 1 achieves fault isolation through spatial coupling using only a single view (V=1), the candidate set size still reaches several percentage points of P due to the inherent hash collisions. For example, with parameters P=65,536, r=1 / 64, and W=2, the candidate set size of a single view is approximately 2,000 partitions. In large-scale distributed systems, if the number of faulty partitions is only 2-5, this size of candidate set contains the vast majority of false positives, making accurate localization difficult. By introducing at least a second independent view and finding the intersection of the candidate sets of the two views, this invention can compress the candidate set size to a very small set (under preferred parameters, as shown in Example 3, it can be compressed to a dozen or so). In contrast, the multi-view mechanism significantly reduces the cost of subsequent accurate repair by improving localization accuracy.

[0062] In summary, the core innovations of this scheme compared to single-view SC-IBLT are reflected in several aspects. First, it adopts a multi-view independent observation mechanism, using V different hash mapping functions (H0, H1, ..., H{V-1}) to map the same partition to completely different segment positions in different views, generating statistically independent observations. Second, it introduces a lightweight SketchSC structure, where the observation probe retains only the count field, omitting keysum and valuesum, thereby reducing network transmission and memory overhead while retaining all the information needed for fault localization. Third, through a drift-free CountScore design, it effectively eliminates background noise caused by A-only / B-only imbalance, making it easier to separate the true difference signal from the noise. Fourth, leveraging the orthogonality of multiple views, it achieves exponential compression of the candidate set size (from r1 to rV) through set intersection, significantly reducing the cost of subsequent accurate repair while maintaining high recall.

[0063] Fifth, an efficient implementation of the reverse mapping table is adopted. The reverse mapping table is pre-computed or dynamically maintained, so that the conversion from hot segment to candidate partition becomes an O(1) table lookup operation, avoiding the overhead of traversing all partitions one by one.

[0064] Option 3: Adaptive gating and scene adaptation based on energy distribution characteristics Technical Background and Problem Statement Building upon Solution 2, this invention further discloses an adaptive gating method. This method predicts the distribution pattern of discrepancies before performing costly fault localization and selects the optimal repair strategy based on the prediction results. This approach addresses the efficiency degradation of Solution 2 under specific noise patterns (such as "diffuse noise"). By introducing a gating mechanism that requires no prior knowledge, it avoids wasting computational resources and significantly improves the robustness and intelligence of the entire data repair system.

[0065] Specific implementation steps Step 1: Calculate the morphological indices of differential energy distribution After obtaining the segment energy scores Es (from Scheme 2) for each view, this method does not immediately perform hotspot screening. Instead, it first treats these scores as a probability distribution describing how the differential energy is distributed spatially. Based on this distribution, one or more "shape metrics" are calculated to quantify their concentration.

[0066] In a preferred embodiment, the preferred morphological index is the Normalized Effective Number of Sections, denoted as Keff / L. Its calculation method is as follows: 1. Energy weight normalization: For any view (or by averaging the results of all views), normalize the energy scores Es of all segments to obtain the energy weight ws of each segment: , This step transforms the segment energy fraction into a probability distribution, such that Σ = 1.

[0067] 2. Calculate the Herfindahl-Hirschman Index (HHI): , The HHI index is widely used in economics and information theory to measure market concentration or uneven distribution. Its value ranges from [1 / L, 1], where: HHI = 1 indicates complete concentration (all energy is concentrated in one segment); HHI = 1 / L indicates complete homogeneity (energy is evenly distributed across L segments).

[0068] 3. Calculate the normalized effective number of segments: The effective number of segments, Keff, is the reciprocal of the HHI, and then normalized using the total number of segments, L. Keff / L = (1 / HHI) / L, The range of this index is [1 / L, 1], and its intuitive physical meaning is "what proportion of the segment is the energy equivalently distributed".

[0069] Physical interpretation of morphological indicators: When differences are highly concentrated in a few partitions (i.e., a "localized explosion" scenario), the energy distribution is extremely uneven, with a large ws and HHI values ​​in a few segments, resulting in a very small Keff / L value. Experiments show that in a typical bimodal fault scenario (where B is the number of fault partitions and dbig is the number of differential bonds within a single fault partition, e.g., B = 2 partitions, dbig = 5000 differential bonds), the median Keff / L is approximately 0.018.

[0070] When the difference is evenly distributed across a large number of partitions (i.e., a "global diffusion" scenario), the energy distribution is nearly uniform, the ws values ​​of all segments are small and close, and the HHI values ​​are also small. Therefore, the Keff / L value will be large. Experiments show that in a diffuse scenario (noiseprob=0.3), the median Keff / L is approximately 0.651.

[0071] The advantage of this metric is that it can automatically generate highly discriminative Keff / L values ​​for both bimodal and diffuse workloads without any prior calibration or explicit input of workload characteristics.

[0072] Experimental Validation of Morphological Indicators: This scheme validated the correlation between the Keff / L index and recognition efficiency through large-scale parameter scanning. Experiments covered different combinations of distribution patterns, fault sizes, and noise intensities. Results showed that in the concentrated mode (low Keff / L value), the recognition efficiency of most sample points was close to optimal, meaning the candidate set highly matched the actual fault partitions. In the diffuse mode (high Keff / L value), the recognition efficiency approached the lower limit, indicating that false positives accounted for the vast majority of the candidate set. This clear distribution pattern provided experimental basis for setting the dual-threshold gating rules.

[0073] Step 2: Apply dual-threshold gating rules for decision-making Based on the morphological index Keff / L, the system employs a dual-threshold (τlo, τhi) gating rule to determine the current difference pattern as one of the following three states and execute the corresponding operation: State 1: Determined to be in "Concentrated Mode" condition: Keff / L < τlo (e.g., τlo = 0.05), decision making: This indicates the presence of a good fault signal, which is within the effective working range of the multi-view positioning method. The system will continue to execute the positioning steps of Scheme 2: Hotspot segment screening is performed, selecting the top t segments with the highest energy scores (where t = L × r); candidate sets Cj for each view are generated; the intersection candidate set C is calculated; precise verification is performed on the partitions in the candidate set to locate the real faulty partitions; finally, the corresponding fault repair protocol (e.g., triggering full data synchronization between replicas) is used to handle the differences within the faulty partitions, so as to prioritize the rapid stabilization of the faulty partitions.

[0074] State 2: Determined to be in "Diffuse Mode" condition: Keff / L > τhi (e.g., τhi = 0.6), decision making: This indicates that the differential energy distribution is relatively uniform, failing to form obvious hotspot areas. Under this mode, the effectiveness of multi-view localization methods will be significantly reduced. The system can choose one of the following solutions: use parameter-optimized standard IBLT for encoding repair, which is suitable for scenarios where the differential scale is relatively controllable; or adopt segmented incremental repair, dividing the data into multiple blocks and processing each block independently.

[0075] State 3: Determined to be in "Gray-Area Mode" condition: τlo ≤ Keff / L ≤ τhi, decision making: This is the boundary region, and the system can adopt several strategies. One approach is to continue with the localization method of Scheme 2, but use a larger diagnostic budget (i.e., a larger hotspot ratio r, for example, increasing it from 1 / 64 to 1 / 32) to increase the detection coverage. Another approach is to introduce auxiliary morphological indicators for secondary judgment, such as log-normalized entropy. (The normalized entropy used to calculate the differential energy distribution is an auxiliary morphological index under the "gray zone mode" in Scheme 3, used to further distinguish the concentrated or diffuse characteristics of the differential distribution. Its value range is [0, 1], close to 0: indicating that the energy is highly concentrated in a few segments (corresponding to "concentrated fault"); close to 1: indicating that the energy is evenly distributed in most segments (corresponding to "diffuse fault").) The energy weight of segment s is obtained by normalizing the segment energy score Es. The energy weights can be represented by the logarithm (usually the natural logarithm or the logarithm with base 2), or the Gini coefficient, to further refine the scene classification. Additionally, the system can execute in parallel, initiating multiple repair schemes simultaneously and using the first completed result.

[0076] This adaptive gating method has been validated in large-scale experiments. In diffuse workloads, ungated systems require time to build multiple views and calculate the energy values ​​of each segment, ultimately resulting in a very small candidate set with limited localization performance. With gating, the system can make decisions based on the Keff / L metric after calculating the first view, thus avoiding unnecessary subsequent calculations, saving approximately 50% of computational costs, and reducing network transmission overhead.

[0077] like Figure 5 and Figure 6 As shown, experimental data demonstrate that the Keff / L-based classifier exhibits relatively stable performance across various workloads. With parameter settings of τlo = 0.05 and τhi = 0.6, the misclassification rate remains below 2% in large-scale experiments (comprising 324,000 trials covering different variance scales and distribution combinations). At the recommended operating point, the tradeoff between the classifier's true positive rate and false positive rate is reasonable. This stability allows the system to automatically adapt to different workloads without frequent parameter adjustments.

[0078] Adaptability without prior knowledge: The entire decision-making process relies solely on the statistics of the differential probe itself (segment energy score Es), requiring no prior information about the true differential distribution or pre-trained data. This allows the system to be directly applied to production environments, adapting to unknown failure modes and dynamically changing workloads, while supporting independent adaptive decision-making in heterogeneous storage systems.

[0079] In summary, this scheme incorporates technical design in several aspects. First, it achieves workload classification without prior knowledge using the Keff / L index, capable of distinguishing different differential distribution patterns. Second, it employs the HHI index, based on information theory, as a morphological indicator, which has a broad theoretical foundation in economics and information theory. Third, it designs a dual-threshold gating architecture, defines clear decision boundaries, and provides strategies for handling boundary cases. Fourth, the morphological indicator has low computational cost, requiring only simple mathematical operations on existing energy scores. Fifth, the method has been validated in large-scale experiments, with a misclassification rate of less than 2% for gating decisions.

[0080] II. To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0081] Important Terminology Explanation: The term "energy" is used repeatedly in this specific implementation. It should be noted that "energy" as used herein does not refer to energy in the physical sense, but rather to the aggregated characteristics of signal strength or count values ​​representing data differences in a statistical table. Specifically, when data differences are numerous and concentrated in a certain segment or partition, its corresponding energy score (e.g., CountScore) will be relatively high, reflecting the "intensity" of the accumulated differences at that location. This terminology borrows from common expressions in spectral analysis and statistical signal processing to facilitate the description of the concentration and spatial distribution characteristics of differences.

[0082] Example 1: Efficient Fault Location in a Binary Fault Scenario Scene setting This embodiment aims to demonstrate the technical effectiveness of the present invention in a "binary failure" or "long-tailed distribution" scenario. This scenario represents a typical failure mode in distributed systems: the vast majority of data partitions remain consistent, with only a very small number of partitions experiencing significant data discrepancies due to configuration errors, software defects, or localized hardware failures. A large-scale distributed key-value storage system is simulated, synchronizing data between two data centers (A and B). Specific parameters are as follows: Partitioning: The system is logically divided into P = 65,536 data partitions.

[0083] Fault configuration: A partial failure occurs, resulting in severe data inconsistencies in B = 2 partitions ("large failure partitions"), with dbig = 5,000 difference keys generated in each large failure partition.

[0084] Background noise: Additionally, a small amount of background noise exists in the system: approximately 10% of the other partitions (about 6,553) each have one or two random, sporadic differences. These small differences are usually caused by inconsistent data write timing or cache refresh latency. The introduction of background noise makes traditional partition checksums and the Markle-Tree method based on this approach unable to distinguish between a major fault and a small number of differences in a partition.

[0085] Total variance size: The total variance size of the entire system is D = 10,000 + 6,553 ≈ 16,553 bond variances.

[0086] Parameter configuration of the method of the present invention To detect and locate the aforementioned differences, in this embodiment, the method described in this invention is configured and applied according to the following parameters: Probe structure parameters (Table 1): Number of segments L = 1024 (Each view contains 1024 segments) Number of units per segment m0 = 12 (Each segment contains 12 units) Number of hash functions k = 3 (each key is mapped using 3 hash functions) Spatial coupling width W = 2 (Each partition covers two consecutive segments) Total number of units M = L × m0 = 12,288 Cell field count (int32) (Only the count field is kept; keysum and valuesum are omitted) Table 1 Diagnostic decision parameters (Table 2): Number of hotspots selected per view t = L × r = 16 (Select the 16 segments with the highest energy) Gating threshold (centralized mode) τlo = 0.05 Gating threshold (diffusion mode) τhi = 0.6 Table 2 Parameter description: It should be noted that the above parameters (e.g., V=2, L=1024, m0=12, r=1 / 64, τlo=0.05, τhi=0.6) are only a preferred embodiment of the present invention in this scenario. Those skilled in the art can flexibly adjust the above parameters according to the actual system's storage budget, network bandwidth conditions, and expected differences, without deviating from the basic technical solution of the present invention. For example, in a system with sufficient memory, L can be increased to improve spatial resolution; in a system with ample network bandwidth, the number of views V can be increased to enhance positioning accuracy; different application scenarios may also require adjustment of the hotspot ratio r and the gating thresholds τlo and τhi.

[0087] Comparison with existing technologies A. Traditional approach (Per-partition Checksum): To locate the fault, checksums for all 65,536 partitions need to be calculated and compared at both ends. If each checksum (such as a SHA-1 hash value) occupies 20 bytes, then the data transfer volume per dimension is: Metadata size = 65,536 × 20B = 1,310,720 B ≈ 1.3 MB.

[0088] In addition, O(P) of storage space is required to maintain these checksum values.

[0089] B. Merkle Tree solution: While using a Merkle tree structure can reduce overall network bandwidth overhead by transmitting only the hash values ​​of the root node and differing branches, this introduces a significant bottleneck in interaction latency. The Merkle tree's layer-by-layer comparison mechanism requires multiple serial communications between the sender and receiver, with the number of rounds proportional to the tree depth (O(log P)). For 65,536 partitions, even with a multi-branch tree structure, several round trips (RTTs) are typically required to locate a specific leaf node. In high-latency network environments spanning geographically dispersed data centers, this multi-round serial interaction significantly increases the total time required for fault location, failing to meet the real-time requirements of rapid fault stabilization in production environments. In contrast, this invention requires only a one-shot probe transmission to complete the location, completely eliminating the latency accumulation caused by multiple rounds of interaction.

[0090] Invention solution: Only two lightweight SketchSC observation probes need to be transmitted (SketchSC: contains only the count field).

[0091] Total metadata: Metadata size = V × L × m0 × sizeof(count) = 2 × 1024 × 12 × 4B = 98,304 B ≈ 96 KB It should be noted that the amount of metadata here refers to the overhead required for a one-way network transfer from one end (e.g., data center A) to the other end (data center B) to complete a diagnosis.

[0092] Data comparison and solution advantage analysis (Table 3): Communication metadata volume Very high (~1.3 MB) Low (small individual packages, but varying total quantity) Extremely low (~96 KB) Bandwidth overhead reduced by approximately 13.5 times. Network interaction rounds 1 time (one-way transmission) Multiple (O(log P) round trips) One-shot Completely eliminate multi-turn interaction delay Storage maintenance overhead O(P) O(P) (tree structure needs to be maintained) O(V . L ) Decoupling storage from the total number of partitions P Location timeliness Limited by the time consumption of big data transmission Limited by network RTT latency Extremely high It combines low bandwidth and zero interactive waiting time Table 3 Conclusion: In this scenario, this invention not only significantly outperforms traditional full checksum schemes in bandwidth efficiency (requiring less than 8% of metadata overhead, i.e., 96KB vs 1.3MB), but more importantly, compared to tree-based synchronization mechanisms such as Merkle Trees, this invention avoids O(log P) multi-round network interactions, achieving "single probe, instant location." This gives this invention an irreplaceable advantage in cross-regional (high latency) or real-time-critical fault "stopping" scenarios.

[0093] Experimental Results and Technical Effects Under the above configuration, after 200 independent Monte Carlo simulation experiments, the statistical average results are as follows: Positioning performance indicators: 1. Candidate set size (CandFrac): Average number of candidate partitions: 50; Compression ratio: CandF rac = 50 / 65,536 ≈ 0.077%; Physical meaning: After the intersection of the two views, the number of partitions that the system needs to further refine and check is reduced from 65,536 to only 50, achieving a compression of approximately 1300 times. The 50 candidate partitions here are the results obtained under a specific experimental scenario containing 10% background noise, including 2 real faulty partitions and 48 false positive partitions caused by both background noise and hash collisions.

[0094] 2. Big Fault Cap Rate (BigCap): succ(bigrecallcount) = 100%, Meaning: In 100% of the experiments, the first two major fault partitions were successfully included in the 50 candidate partitions, indicating that the method is very reliable in capturing the "primary contradiction".

[0095] 3. Overall Differential Recall: Recall = 41.8% Meaning: The candidate set covers an average of 41.8% of the total differences. This indicates that in addition to 100% of the major faults, some background noise (approximately 0.8% of the noise differences) was also successfully captured.

[0096] 4. Recognition efficiency (R / C ratio): R / C = Recall / CandF rac = 0.418 / 0.00077 ≈ 540, Meaning: The improvement in location efficiency compared to random screening. This means that the location efficiency of this method is 540 times that of random sampling for partition inspection. With the same computational budget, this method can capture 540 times the quality of variance.

[0097] Accuracy verification: In these 200 experiments, the following accuracy verifications were also performed: False positive rate (clean partitions included in the error): approximately 48 on average (i.e., 50-2=48). This is due to the "halo effect" caused by the single-view window width W=2, which is an acceptable trade-off in SC-IBLT design. False negative rate (missed fault partitions): 0%, 100% of major faults were successfully located; Subsequent processing flow and end-to-end performance Based on the above location results, the subsequent repair process of the system became extremely efficient, forming a two-stage strategy of "prioritizing the repair of major faults and subsequently cleaning up residual noise": Phase 1: High-priority fault repair 1. Candidate Partition Verification: For the 50 identified candidate partitions, simultaneously obtain the partition's metadata (such as the total number of objects or updateversion). The additional overhead of this step remains minimal. Additional overhead = 50 × (8B Metadata) ≈ 400B.

[0098] 2. Difference Classification and Locking: Through combined analysis of checksums and metadata, candidate partitions are accurately classified. Major Fault Lockout: For partitions with mismatched checksums and huge differences in the number of objects (or extremely high energy score Es in Scheme 2), they are identified as "major fault partitions" (i.e., the two most severely affected areas).

[0099] Noise filtering: For partitions with mismatched checksums but identical object counts and low energy scores, they are classified as "background noise partitions" and left for processing in the second stage.

[0100] 3. For this major fault and high-energy partition, immediately implement a high-priority repair strategy: Option A: Version rollback (if the fault is software-related); Option B: Service Degradation (Temporarily disable external services for this partition); Option C: Full snapshot synchronization (the safest but most expensive option).

[0101] In this scenario, the cost of snapshot synchronization is: 2 × 5,000 keys ≈ only 10,000 keys need to be repaired, which is 1300 times lower than the cost of synchronizing the entire 65,536 partitions.

[0102] Phase Two: Low-Priority Diffuse Noise Cleanup 4. For the 58.2% of diffuse noise (approximately 9,636 bonds) that was missed and not located, as well as the low-energy partitions from the first stage, a low-cost "mopping-up" cleanup can be performed using a global, more reparative standard IBLT protocol: Global IBLT parameter settings: - M = 1.3 × 9,636 ≈ 12,527 units; - Storage overhead: 12,527 × 8B ≈ 100 KB.

[0103] Note: In practical systems, the exact value of diffuse noise is usually not known. It can be estimated based on daily experience (e.g., 1-2% of the total). Higher redundancy can be provided if engineering permits.

[0104] End-to-end cost summary: Probe transfer 96 KB + candidate set refinement 1 KB + global IBLT 100 KB = total cost 200 KB. In comparison (Table 4): Compared to full IBLT repair (M=1.3×16,553≈ 21,500 units, 172KB) Save 92% Table 4 Summary of the significance of this embodiment This embodiment fully demonstrates the advantages of the present invention in binary fault scenarios: 1. Extremely low pre-detection cost: Quickly locate problems with a probe of only 96KB, avoiding expensive full scans. 2. High fault location accuracy: Accurately locates two large fault partitions with a probability of over 95%. In addition, Recall=41.8% means that it captures about half of all the differences at the same time.

[0105] 3. Intelligent repair strategy: Dynamically selects repair plan based on positioning results, achieving rapid hemostasis in a "firefighting" manner. 4. Excellent end-to-end performance: The overall cost (probe + detailed inspection + global repair) is only 14% of that of traditional solutions. Such performance indicators indicate that the present invention is particularly suitable for production environments with "concentrated faults and the need for rapid hemostasis".

[0106] Example 2: Adaptive Degradation in a Diffuse Noise Scene Scene setting This embodiment aims to demonstrate the behavior and technical effectiveness of the method of the present invention in a "diffuse noise" scenario. Unlike the "binary fault" in Embodiment 1, in this scenario, data differences are evenly distributed on a small scale across a large number of partitions, without obvious "severely affected areas." This scenario is common in production systems in the following ways: Slow drift caused by long-term, low-intensity cross-regional delay; a situation where multiple minor faults overlap but none of them are serious. The convergence process of the distributed algorithm in the eventual consensus phase. The simulation parameters are as follows: Total number of partitions: P = 65,536 (same as Example 1); Difference distribution: Differences are evenly distributed across 30% of the partitions in the system (approximately 19,660 partitions), with each partition containing 1 to 2 random difference keys; Fault configuration: There are no "large fault partitions" (B = 0), meaning no single partition has a difference size exceeding the threshold; Overall variance: The total variance across the entire system is approximately D ≈ 19,660 × 1.5 ≈ 29,490 bonds, which is about 1.78 times that of Example 1 (16,553 bonds). The characteristics of differential distribution: the differential "energy" is evenly distributed in multiple segments and will not form obvious "peaks" in any segment.

[0107] Parameter configuration To ensure a direct comparison with Example 1, this example uses the exact same probe structure and diagnostic decision parameters: Probe structure parameters (Table 5): Number of segments L = 1024 (Same as Example 1) Number of units per segment m0 = 12 (Same as Example 1) Number of hash functions k = 3 (Same as Example 1) Spatial coupling width W = 2 (Same as Example 1) Table 5 Diagnostic decision parameters: The hot zone ratio r = 1 / 64 (same as in Example 1); The gating threshold (centralized mode) τlo = 0.05; The gating threshold (diffusion mode) τhi = 0.6.

[0108] Prediction and triggering of gating mechanisms The system first applies Scheme 3 to predict the pattern of the differential distribution. The detailed steps are as follows: Step 1: Calculation of technical indicators The system calculates the segment energy fraction Es for each view based on the differential probe, and then calculates the morphological index Keff / L of the energy distribution. The specific calculation process is as follows: 1. Energy weight normalization: This converts the energy fraction into a probability distribution. 2. Calculate the HHI index: .

[0109] 3. Calculate the number of valid segments: Keff / L = (1 / HHI) / L.

[0110] In the 200 independent experiments in this simulation, the statistical results of the Keff / L values ​​obtained are as follows: median: 0.651, mean: 0.658, standard deviation: 0.042.

[0111] Physical meaning: Keff / L ≈ 0.65 indicates that the differential energy is equivalently "spread" over about 65% of the entire set, meaning that the energy distribution is nearly perfectly uniform.

[0112] Comparative analysis (Table 6): Example 1 (Binary Fault) 0.018 Energy is extremely concentrated, ~1.8% of the segment. Example 2 (Diffuse Noise) 0.651 Energy is highly uniform, with ~65% of the segments being highly uniform. Discrimination 36× difference Automatic classification is extremely robust. Table 6 The 36-fold discrimination ratio shown in this table fully demonstrates the powerful discriminative ability of the morphological index Keff / L. There is a clear correlation between this index and recognition efficiency: a high Keff / L value corresponds to low recognition efficiency, and vice versa. The Keff / L ≈ 0.65 obtained in Example 2 indicates that the system is in the "diffuse mode" region, where the recognition efficiency is close to the lower limit, fully verifying the correctness of the gating decision in this example.

[0113] Step 2: Triggering the gating rules The obtained Keff / L value (0.65) was compared with the system's preset "diffusion mode" high-order threshold τhi = 0.6: Judgment condition: Keff / L > τhi ? Result: 0.65 > 0.6✓ The condition is met.

[0114] Gating decision: Triggering "diffuse mode", the system determines that the current scene is not suitable for multi-view intersection localization.

[0115] Step 3: Automatic System Degradation Based on the gating decision, the system immediately suspends the subsequent "hotspot segment filtering" and "candidate set intersection" operations, and instead adopts one of the following degradation strategies: Option A (Recommended): Construct a standard IBLT covering the entire dataset (eliminating the need for partitioning and localization) to directly perform global difference repair. Global IBLT parameters: Mglobal = 1.3 × D = 1.3 × 29,490 ≈ 38,337 units. Storage overhead: 38,337 × 8B ≈ 307 KB; Option B: Trigger snapshot synchronization between multiple data centers to ensure consistency in a simple and brute-force manner; Option C: Phased incremental repair, dividing the full data into multiple independent small blocks, applying standard IBLT to each block, and finally summarizing the results.

[0116] Hypothetical scenario comparison: Forced location verification (validating the necessity of gating) To fully verify the accuracy of the gating mechanism's prediction, we conducted a reverse analysis: assuming the gating mechanism was forcibly bypassed, the multi-view intersection localization process in Example 1 was continued. The results are as follows (based on the average of 200 independent simulations): Positioning performance indicators: Candidate set size (CandF rac): 0.095%; Overall differential recall (Recall): 0.14%; Recognition efficiency (R / C Ratio): 0.14% / 0.095% ≈ 1.47.

[0117] Key observations: 1. R / C ≈ 1.47, close to the R / C of 1.0 for random selection. This indicates that the positioning effect is almost indistinguishable from random sampling and partitioning. The positioning is ineffective and wastes computing resources.

[0118] 2. The candidate set is extremely small (0.095%) but the recall is extremely low (0.14%). Although the candidate set was compressed to only 62 partitions, it contained very little difference information.

[0119] 3. Comparative Example 1: R / C = 540 Example 1 (Centralized Mode): R / C = 540, extremely efficient positioning; Example 2 (Diffusion Mode): R / C = 1.47, positioning completely failed; The efficiency decreased by about 370 times, illustrating the fundamental difference between the two scenarios.

[0120] System behavior and technical effects In this embodiment, the complete behavior of the present invention is executed as follows: 1. The system constructs and differentiates V=2 views into a multi-view probe; 2. Calculate the segment energy fraction Es for each view (total time < 1ms); 3. Calculate the morphological index Keff / L = 0.65 (total time < 0.1ms); 4. Apply gating rule: Keff / L (0.65) > τhi (0.6) ✓ Trigger; 5. The system automatically reverts to the global IBLT repair protocol (location is no longer performed); 6. Build a global IBLT for global repair.

[0121] Comparative Analysis (Table 7): Complementarity of the Two Examples Keff / L 0.018 0.651 36 times greater discrimination, reliable automatic classification R / C 540 1.47 The efficiency difference is 370 times, making a switch necessary. Location effectiveness ✓ Extremely high X is invalid Option 2 is designed specifically for centralized mode. Repair strategy Positioning + Fine Repair Global IBLT The Importance of Choosing a Strategy Gating Necessity No (It will automatically enter the centralized process) Yes (to prevent resource waste) Value of Option 3 Table 7 This embodiment strongly demonstrates the adaptability and robustness of the invention: 1. Comprehensive scene coverage: Solution 2 (multi-view localization) performs well in cases of concentrated failures, while Solution 3 ensures correct decision-making in diffuse scenarios; 2. Seamless strategy switching: The system can automatically select the most suitable repair solution in milliseconds, without requiring users to manually adjust parameters; 3. Production-grade robustness: This invention not only handles "ideal cases", but more importantly, it also performs well in "non-ideal cases", avoiding the dilemma of "failure in specific scenarios" that traditional methods may fall into; 4. Practical value of morphological indicators: Keff / L, as a simple yet powerful morphological measure, provides decision-making basis for the entire system with extremely low computational cost (<1ms), embodying the elegance of "informatics"; This complementary design enables the invention to adapt to diverse failure modes in practical deployments, making it a robust and adaptive production-grade fault detection and repair solution.

[0122] Example 3: Parameter Sensitivity Analysis and Tuning Guide Purpose of the Implementation Example This embodiment aims to analyze how several core design parameters of the present invention, including the number of segments L, the number of views V, and the coupling width w, affect the efficiency and accuracy of fault location. Understanding the role of these parameters is crucial for performance optimization and cost control in different application scenarios. The analysis in this embodiment is conducted under the "binary fault" scenario described in Embodiment 1 (P=65,536, B=2, dbig=5,000).

[0123] Parameter Analysis Overview This analysis covers the following parameter space: Analysis 1: The Impact of the Number of Segments L – A Trade-off Between Resolution and Cost Parameter mechanism: L determines the probe's "spatial resolution". The larger L is, the weaker the tendency for the same amount of fault energy to be distributed across more segments, thus the more pronounced the "peak" of the fault signal in a local segment will be. At the same time, the larger L is, the larger the probe's memory overhead (approximately O(L)).

[0124] Experimental design: With other parameters fixed (V=2, W=2, m0=12, k=3, r=1 / 64), the performance of L on {256, 512, 1024, 2048} was scanned. 200 independent simulations were performed for each L value, and the average results were statistically analyzed.

[0125] Experimental results and phenomena (Table 8): 256 1 / 32 ~400 0.61% 24 KB Insufficient resolution, requires larger resolution. 512 1 / 32 ~200 0.31% 48 KB Performance began to improve 1024 1 / 64 ~50 0.077% 96 KB Best cost performance 2048 1 / 64 ~45 0.069% 192 KB Accuracy improvement <2%, overhead doubled Table 8 Key observations: 1. Insufficient resolution (L=256): When L is small, the energy of multiple partitions is more likely to "alias" in the same segment, resulting in the energy peaks not being "sharp" enough; In order to still capture all large faults, a large hot zone ratio r is needed (e.g., r=1 / 32), which will absorb a lot of background noise, resulting in an oversized final candidate set.

[0126] Physical analogy: It's like taking a picture with a low-resolution camera; even if you increase the exposure, it's difficult to capture details clearly.

[0127] 2. Optimal resolution range (L=1024): The probe resolution is significantly improved, and the energy of the fault partition can be more concentrated in a few segments; Only a small hot zone ratio r (e.g., r=1 / 64) is needed to capture all major faults while avoiding excessive noise absorption; The candidate set size was reduced from 400 when L=256 to 50, a compression of 8 times; This is the dessert area that achieves a balance between "high precision" and "low overhead".

[0128] 3. Excessive resolution (L=2048): Although the resolution continues to improve, the marginal improvement in positioning accuracy is small (the candidate set only slightly decreases from 50 to 45, an improvement of only 6%). The probe's memory overhead doubled (from 196KB to 392KB). The diminishing returns effect is obvious and does not conform to engineering principles.

[0129] Recommended conclusion: Under the parameter configuration of this experiment (m0=12, V=2, P=65,536), L=1024 is the optimal choice between achieving high accuracy and low cost. Similar cost-effectiveness analyses can be conducted based on this conclusion under other parameter scenarios.

[0130] Parameter generalization explanation: The above conclusions are based on this specific parameter configuration and fault scenario (binary fault: B=2, dbig=5,000). In practical applications, if the partition size P, the number of faulty partitions B, or the fault density dbig changes, the optimal value of L may be adjusted accordingly. Those skilled in the art can refer to the methods and ideas of this analysis to optimize the parameters for specific system configurations, and are not limited to the specific values ​​given in this embodiment. For example, in systems with a larger total number of partitions (e.g., P=2^20), L may need to be increased accordingly; in scenarios with higher fault density, the combination of L and W may need to be adjusted.

[0131] Analysis 2: The impact of the number of views V – exponential compression of the candidate set Parameter Mechanism: V is the core parameter for achieving high-precision positioning in this invention. Each additional view provides the system with an independent, statistically uncorrelated fault observation perspective. The intersection of candidate sets from multiple views can exponentially reduce the candidate space, which is the fundamental advantage of this invention compared to single-view IBLT.

[0132] Experimental design: With L=1024, r=1 / 64, W=2, m0=12, k=3 fixed, the candidate set size and accuracy were compared when V was 1, 2, and 3. 200 independent simulations were performed for each V value.

[0133] Experimental results and phenomena (Table 9): 1 ~1,540 2.4% 1 × 100% Single view, halo too large 2 ~13 0.02% ~118× 100% Order of magnitude improvement 3 ~2.3 0.0035% ~670× 100% Approaching the lower bound B=2 Table 9 Key observations: 1. Limitations of a single view (V=1): Theoretical analysis: In a single view, the candidate set size is approximately P × (W × r) = 65,536 × (2 × 1 / 64) = 2,048 partitions. Experimental results: Approximately 1,540 partitions (slightly better than the theoretical result, as some hotspots are not obvious). Problem: Approximately 1,500 clean partitions are still mixed into the candidate set, resulting in inaccurate localization.

[0134] 2. Qualitative change in dual-view structure (V=2): Theoretical analysis: The intersection of the two views should give P × (W×r)^2 = 65,536 × (1 / 32)^2 ≈ 64 partitions.

[0135] Experimental results: Approximately 13 partitions (better, because there is a negative correlation optimization effect).

[0136] Improvement: Reduced from 1,540 to 13, achieving a compression ratio of 118x.

[0137] 3. Diminishing returns on three-view drawings (V=3): Theoretical analysis: P × (W × r)^3 ≈ 2 partitions (close to the lower bound of B=2).

[0138] Experimental results: Approximately 2.3 partitions.

[0139] Improvement: Only 56% more compression compared to V=2 (from 13 to 2.3).

[0140] Cost: A third view probe needs to be built, increasing network traffic by 50%.

[0141] in conclusion: V=2 is the optimal choice for most applications, providing 118x candidate set compression while only doubling network and storage overhead (compared to a single view). V=3 is suitable for scenarios with extremely high positioning accuracy requirements, but requires accepting an additional 50% overhead. V≥4 is not recommended because network transmission costs increase linearly with each additional view, while the marginal benefit of candidate set compression decays exponentially.

[0142] Parameter selection flexibility: The above recommendations are based on the binary fault scenario of this embodiment. For different application scenarios, the optimal V value may need to be adjusted. For example, in very large-scale systems (P > 2^20) or scenarios with a larger number of fault partitions, more views may be needed to ensure positioning accuracy; while in edge systems with extremely limited network bandwidth, only V=2 may be used, although accuracy may decrease. Those skilled in the art should flexibly select the number of views according to specific system constraints and accuracy requirements.

[0143] Analysis 3: The Influence of Coupling Width w and the Scaling Law of Candidate Sets Parameter mechanism The coupling width *w* determines that a partition will affect *w* consecutive segments (within a window of [base, base+W-1]). This parameter has a dual effect: Negative: The larger w is, the larger the "halo" corresponding to each partition in the single view, resulting in an overly large candidate set for the single view. Front: w, V, and r together determine the final multi-view intersection compression effect. Experimental Design and Scaling Law Discovery: In numerous experiments with "binary fault" scenarios, we statistically analyzed the relationship between the candidate set compression ratio CandFrac and (W, r, V), and discovered an empirical scaling law: CandF rac ≈ c · (W × r)^V, Where c is a constant close to 1 to 3.5, depending on the specific characteristics and distribution properties of the hash function.

[0144] Validation data (Table 10) (L=1024, m0=12, V=2 fixed): 1 1 / 64 0.000244 0.00061 2.5 normal 2 1 / 64 0.000977 0.00077 0.79 Underestimating (negative correlation effect) 3 1 / 64 0.002197 0.0022 1.0 accurate 4 1 / 64 0.003906 0.0038 0.97 accurate Table 10 1. The universality of scaling laws: The formula demonstrates good predictive power under different (L, V) combinations, and it provides a powerful budget prediction model: given a (W, r, V) configuration, it can accurately estimate the number of partitions required for diagnosis.

[0145] 2. The complementary relationship between w and V (with a fixed total probe size): Scenario: Assume the total storage budget is fixed at 200KB, corresponding to L×m0×V = constant. Option A: W = 1, V = 2, L = 1024 Candidate set: ~650 partitions (when r = 1 / 64).

[0146] Option B: W=4, V=4, L=512 (same total number of cells) Candidate set: ~2 partitions (when r = 1 / 64).

[0147] Conclusion: Although the single-view halo is larger (4 times) with W=4, the exponential compression (r2 → r4) with V=4 is more powerful, resulting in a final candidate set that is 330 times smaller! 3. Optimization space of nonlinear compression: Using CandF rac ≈ c ·(w×r) V This formula can find the (W, V, L) combination that minimizes the candidate set under the constraint of a fixed total probe size. This involves an optimization problem: minimize CandF rac = c · (w×r)V Subject to: V × L × m0 = Mbudget (total number of cells is fixed) W, V, L, r ∈ [feasible region] The solution shows that, when the budget is sufficient, one should tend to increase V (to obtain exponential compression gains) rather than simply increase L (to obtain only logarithmic resolution improvement).

[0148] Recommendation Conclusion 1. For standard applications (P~65,536, B~2-10): Recommended configuration: W=2, V=2, L=1024 This offers the best general performance and ease of implementation.

[0149] 2. For storage-constrained scenarios: Recommended strategy: Decrease L (e.g., L=512) while increasing V (e.g., V=3 or 4). Although the resolution of a single view decreases, the exponential compression of the intersection of multiple views often compensates for this loss.

[0150] 3. For extremely large-scale scenarios (P > 1 million): Recommended strategy: Increase w (e.g., W=3 or 4) and increase V (e.g., V=3 or 4). Increasing the window width can improve the "noise robustness" of the candidate set (especially under large P).

[0151] Scope of application for recommended parameters: The recommended configurations (W=2, V=2, L=1024, etc.) and parameter tuning guidelines mentioned above are based on the simulated fault scenarios and system configurations in this embodiment. In actual deployments, different application scenarios (such as network I / O intensive, storage capacity-limited, latency-sensitive, etc.) may require different parameter trade-offs. Those skilled in the art should make corresponding customized adjustments based on specific hardware conditions, performance indicators, cost constraints, and other factors, and are not limited to the specific values ​​given in this embodiment. The core technical solutions of this invention (spatial coupling, multi-view intersection, adaptive gating) have sufficient flexibility and versatility to adapt to various parameter selections.

[0152] III. Beneficial Effects of the Invention Compared to existing technologies, this invention achieves significant technological advancements in several key dimensions by introducing spatial coupling design and a multi-view observation mechanism. The following details the beneficial effects of this invention compared to mainstream existing technologies: Beneficial effect 1: Significant reduction in metadata overhead Problem Background: In large-scale distributed storage systems, when the number of partitions P reaches tens of thousands or even millions, the traditional "full checksum" method needs to maintain and transmit metadata of scale O(P), which becomes a bottleneck of the system.

[0153] This invention decouples the metadata size from the number of partitions P, and instead relates it to the probe's geometric parameters (V, L, m0), with a size of O(V×L×m0), which is much smaller than P.

[0154] Specific data comparison (from Example 1, P=65,536): Traditional Per-partition Checksum: Metadata size = P × 20B = 65,536 × 20B = 1,310,720 B ≈ 1.3 MB Features: It grows linearly with P. When P increases to 1 million, the metadata expands to 20MB.

[0155] Invention solution: Metadata size = V × L × m0 × 4B = 2 × 1024 × 12 × 4B = 98,304 B ≈ 96KB Features: Constant size, independent of P. Even if P reaches 1 million, it still only requires 96KB.

[0156] Compression ratio: 96KB / 1.3MB ≈ 7.4% (i.e., 13.5 times compression).

[0157] Benefits in extremely large-scale scenarios: Assume P = 1,000,000 (1 million partitions): Traditional Checksum: 20MB of metadata, resulting in huge pressure on transmission and storage.

[0158] This invention: 96KB metadata, transmission latency <100ms (even on gigabit networks).

[0159] Benefits: Enables real-time probing of large-scale systems.

[0160] Beneficial effect 2: Natural fault isolation capability Problem Background: Standard IBLT uses a random hashing mechanism: the k hash positions of each key are uniformly and randomly distributed across a tablespace of M units. This leads to a serious problem: Scenario: Partition A experiences an extremely severe failure (e.g., a difference of millions). Consequences of traditional IBLT: - The difference keys of A are randomly distributed throughout the table; - Almost all cells in the table are "contaminated" (count value is extremely high); - The detection capabilities of other partitions B, C, and D are completely disabled; - The system is in a "global blind spot" and cannot locate A itself.

[0161] Advantages of this invention: This invention uses a spatial coupling mechanism to limit the influence of each partition to a continuous local window: Scenario: A million-level failure occurs in the same Partition A. Results of the invention: - All differences in A are constrained within the continuous window of [base, base+W-1]; - Only the cell in this window is contaminated; the other windows remain clean. Partitions B, C, and D can still be correctly detected; - The system achieves "compartmental isolation," meaning that local failures do not affect the overall system.

[0162] Beneficial effect 3: Excellent CPU cache affinity Background: In high-throughput data centers (millions of QPS), CPU cache performance is critical. The standard IBLT memory access pattern: When a key is inserted, k hash locations need to be accessed. These locations are generated by k independent hash functions and are usually scattered in different areas of memory, such as 0x100, 0x9000, 0x500 (highly discrete addresses).

[0163] Consequence: Under a load of 10M QPS, cache misses will become a serious performance bottleneck.

[0164] Advantages of this invention: The continuous window mapping mechanism of this invention produces natural address locality: The memory access mode of this invention is as follows: when a key is inserted, all k positions fall within the continuous range of [base×m0, (base+W)×m0). For example, if base=100, W=2, m0=12, then the range is [1200, 1464), and all k positions are within this continuous interval.

[0165] Beneficial Effect 4: Unique ability to compress candidate sets exponentially. Background: In single-view IBLT, due to hash collisions, the size of the candidate set is often proportional to the total number of partitions. Candidate set size for single-view IBLT: CandF rac ≈ W × r Example: When P=65,536, W=2, r=1 / 64 CandFrac ≈ 2 × (1 / 64) ≈ 3.1% The candidate set size is approximately 65,536 × 3.1% ≈ 2,000 partitions.

[0166] Problem: The 2,000 candidate partitions still need to be further examined one by one, which is costly.

[0167] Advantages of this invention: The multi-view intersection mechanism of this invention introduces a capability that is completely absent in the prior art: Exponential compression law: CandF rac ≈ c · (w · r) V ; Dual-view configuration (V=2): Theoretical compression: (1 / 64) 2 = 1 / 4,096 ≈ 0.024%, Experimental results: 0.02% ← Achieved the theoretical value! Candidate set size: 0.02% × 65,536 ≈ 13 partitions.

[0168] Three-view configuration (V=3): Theoretical compression: (1 / 64) 3 = 1 / 262,144 ≈ 0.0004%, Experimental results: 0.0035% (slightly higher than theoretical, due to the halo effect). Candidate set size: ~2-3 partitions ← Approaching the lower bound of the number of faulty partitions B=2.

[0169] Benefit 5: Robustness Advantages of Goal Shift and Two-Stage Repair Strategy Core Innovation: A Paradigm Shift from "Precise Recovery" to "Fault Location" A fundamental shift in this invention compared to traditional IBLT lies in the redefinition of the repair objective. Traditional IBLT aims to fully recover all differing keys, which can lead to table overflow and decoding failures when the difference scale is large. This invention proposes a fundamentally different solution: focusing only on efficiently locating the "faulty partitions" that generate large-scale differences, rather than attempting to recover all differing keys.

[0170] Implementation of the two-stage repair strategy This shift in objectives enables the present invention to implement a two-stage repair process. The first stage uses a lightweight probe (only 96KB) to quickly identify faulty partitions, and then employs rapid repair strategies (such as direct full snapshot synchronization, version rollback, etc.) to handle the vast majority of differences. The second stage, after excluding faulty partitions, cleans up residual, small-scale diffuse differences. At this stage, a more precise repair scheme (such as standard IBLT) can be used because the data volume is significantly reduced, making the cost relatively manageable.

[0171] Operational feasibility This phased approach aligns with the real-world operational scenarios of large-scale distributed systems. In actual failures, system problems typically stem from data inconsistencies in a few partitions, rather than global discrepancies. Compared to traditional "one-size-fits-all" repair methods (either full synchronization or full IBLT encoding / decoding), this invention's two-phase solution offers the following advantages: First, prioritizing the location and handling of faulty partitions reduces unnecessary computational and network overhead. Second, the system can select appropriate repair strategies based on the actual distribution pattern of discrepancies (through the adaptive gating mechanism in Scheme 3), rather than being forced to adopt a fixed approach. Third, when a phase encounters difficulties, the system can employ a degradation strategy to prevent the collapse of the overall repair process. Finally, system administrators can flexibly adjust the aggressiveness of the repair strategy based on the severity of the fault.

[0172] For example, in a scenario where a hardware failure occurs in a partition, resulting in a large amount of data inconsistency, traditional solutions require waiting for a complete difference recovery process before the location of the failure can be determined; while this invention can locate the faulty partition within seconds and immediately trigger failover or rapid synchronization, shortening the overall recovery time.

[0173] Summary table of beneficial effects (Table 11)

[0174] Based on the above five beneficial effects, the key dimensions of this invention compared with existing technologies are as follows: metadata volume O(P) has a size that increases linearly with the number of partitions. O(M), related to the number of units O(V·L·m0), decoupled from P, has a constant scale. Example (P=65,536) 1.3 MB 96 KB ~ Hundreds of KB 96 KB Fault Isolation ✓ Natural isolation (partition-by-partition verification) X Global Contamination (Random Hash) ✓ Spatial coupling isolation (localization) CPU cache affinity N / A Random access (poor caching performance) Continuous access Candidate set compression N / A Uncompressed (full data rollback required) Exponential compression, (W·r)^V scaling law Adaptability X (Fixed Plan) none ✓ Option 3 Gating, Automatic Adaptation Table 11

[0175] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data difference detection method based on spatial coupling, characterized in that, Includes the following steps: Logical data partitioning: The data items in the first and second datasets are logically divided into multiple partitions based on their corresponding partition IDs; Construct a difference detection table: Construct a difference detection table containing multiple consecutive sections, wherein each consecutive section contains several cells, and each cell contains at least one count field for recording difference statistics; Spatial coupling mapping from partition to segment: For any partition, it is associated with a local window consisting of several consecutive segments in the difference detection table through a preset deterministic mapping rule, and continuous mapping is performed to ensure that all difference information of the same partition is constrained within the window. Key-value encoding: For each data item's key within the partition, at least one hash function is used to map the key to a cell within the local window associated with its partition, and corresponding counting or aggregation operations are performed on the cell. Generate difference statistics: Based on two difference detection tables constructed according to the first dataset and the second dataset respectively, difference statistics are obtained through difference operation and corresponding difference tables are generated. The difference statistics reflect the distribution of difference signals within each local window.

2. The method according to claim 1, characterized in that, In addition to the counting field, the unit also includes a key checksum field and a value checksum field, which are used to invert and recover specific differential key-value pairs during the differential data recovery phase after fault location.

3. The method according to claim 1, characterized in that, In the spatial coupling mapping step, the relationship between the partition identifier pid of the partition and the starting segment index base of its mapped contiguous window is as follows: base = H(pid) mod (L - W + 1), Where H is a preset hash function, L is the total number of segments, and W is the window width, to ensure that the end position of the window does not exceed L-1.

4. A fault partition identification method based on multi-view intersection, characterized in that, Includes the following steps: Constructing multi-view observation probes: Construct at least V independent observation probes, each of which is an observation view, i.e., a difference table, constructed using the method described in claim 1, and the local window positions mapped in different observation probes for the same partition are independent of each other; Calculate the segment energy feature score for each view: For each segment of each observed view, calculate a feature score, which is used to characterize the statistical significance of the differential data contained within the segment; Hotspot filtering: In each observation view, the hotspot segments with the highest feature scores are selected based on the sorting of the feature scores. Candidate set generation and intersection: Based on the mapping relationship between hotspot segments and partitions, determine the candidate partition ID set for each observation view, and perform an intersection operation on all the candidate partition ID sets to obtain the final fault partition candidate set. The intersection operation is used to compress the size of the candidate set.

5. The method according to claim 4, characterized in that, The cells of the V independent observation probes contain only a count field, while omitting the keysum and valuesum fields.

6. The method according to claim 4, characterized in that, Mapping independence between views is achieved by applying different affine shuffle functions to each view j based on the partition identifier pid: Pid_shuffled_j = (aj * pid + bj) mod P, where aj is coprime to the total number of partitions P, bj is a random or pseudo-random offset, and P is the total number of partitions.

7. The method according to claim 4, characterized in that, In the step of calculating the segment energy feature score for each view, a de-drift correction mechanism is used to determine whether there is a true set of differences within the segment. This mechanism includes the following calculation steps: First, calculate the statistics of cell counts within the segment. For any segment s, define: First-order statistic (net sum): , Second-order statistic (sum of squares): , in, The value of the count field in the i-th unit of the s-th segment is the key in the source set A that is mapped to that unit; Then, the feature score Es is calculated through drift correction: , This represents the number of units.

8. The method according to claim 4, characterized in that, The candidate generation and intersection step also includes a pre-calculated or dynamically generated reverse mapping table for back mapping segment indexes to all partition IDs that may cover the segment; the table is constructed as follows: for each view j and each partition pid, calculate all segments covered by its corresponding window, and add the pid to the partition list corresponding to these segments.

9. An adaptive data discrepancy repair decision method, implemented based on the method of claim 4, characterized in that, Includes the following steps: The morphological indices for calculating differential energy distribution include: Obtain the feature score distribution of each segment in the difference detection table, and calculate a morphological index to measure the concentration of feature scores based on this distribution. The morphological index reflects the distribution characteristics of differences in each segment, and its value ranges from 0 to 1, where close to 0 indicates that the feature scores are extremely concentrated in a few segments, and close to 1 indicates that the feature scores are evenly distributed in a large number of segments. The morphological index includes the normalized effective number of segments calculated based on the Herfindahl-Hirschman index. Decision-making is performed using a dual-threshold gating system, including: The morphological indicators are compared with one or more preset thresholds, and subsequent data repair strategies are automatically selected based on the comparison results. If the morphological indicators are below the lower threshold, it is determined to be a concentrated mode (differences are concentrated in a few partitions), and the system executes a precise fault partition location process; If the morphological indicators exceed the upper limit threshold, it is determined to be a diffuse mode (differences are evenly distributed across most partitions), and the system degrades to the global repair protocol or full data synchronization; If the morphological indicators are between the upper and lower thresholds, the system can adopt a gray zone strategy, such as enhancing observation (more views) or adjusting the filtering parameters.

10. A data synchronization system, characterized in that, include: Probe construction module: used to perform the method as described in claim 1 at the source end and the target end to construct difference detection tables respectively, and optionally perform the multi-view construction as described in claim 4; Differential and Recognition Module: Used to receive the difference detection table, perform differential operations, and execute the method as described in claim 4 to identify the candidate set of fault partitions; Repair decision module: used to perform corresponding subsequent data repair operations based on the candidate set of fault partitions, or based on the decision output by the adaptive data difference repair decision method as described in claim 9, including performing precise checksum, version rollback and / or full snapshot synchronization on the candidate partitions.

Images