Near-memory processing unit and data lake cleansing system oriented to data cleansing

Abstract

The present application relates to the technical field of data cleaning, and more particularly to a near-storage processing unit and a data lake cleaning system for data cleaning, which integrates processing logics such as signature generation, band segmentation and bucket mapping, Theta Sketch coding, Odd Sketch construction and cleaning digestion in the storage medium controller layer, generates SimHash fixed-length binary signatures for original data records locally at each storage node, establishes a band bucket mapping table according to band segmentation to screen local same-bucket near-duplicate candidate pairs, constructs a Theta Sketch structure in units of band identifiers of each band segment position and uploads it to a coordination node to perform cross-node set intersection operations to find cross-node near-duplicate candidate pairs, and then refines and screens the near-duplicate record pairs through an Odd Sketch bit vector and performs field-level accurate comparison to gradually confirm the verified near-duplicate record pairs. The present application realizes in-situ execution of cleaning calculation at the storage end, and greatly reduces data migration bandwidth consumption.

Description

Technical Field

[0001] This invention belongs to the field of data cleaning technology, specifically relating to near-memory processing units and data lake cleaning systems for data cleaning. Background Technology

[0002] Existing data cleaning solutions primarily involve moving raw data from storage nodes to host computing nodes for processing. Specifically, the host initiates a data read request, transferring all records in the target data partition to host memory via the storage interconnect network, where the host processor executes a deduplication algorithm. At the deduplication algorithm level, several mature approximate deduplication techniques exist. The SimHash algorithm compresses high-dimensional data into fixed-length binary signatures through random hyperplane projection, using the Hamming distance between signatures to measure the similarity of the original records. It has advantages such as low computational complexity and fixed signature length, making it suitable for fast approximate comparison of large-scale data. The MinHash algorithm samples the minimum hash value of elements in a set based on random permutations, estimating Jaccard similarity through signature collision probability, and is widely used for document-level near-duplicate detection. Bloom filters and their variants map elements into bit arrays using multiple hash functions, achieving probabilistic determination of set members with minimal space overhead, and are often used as pre-filters in deduplication pipelines. In a distributed environment, Theta Sketch, as a probabilistic data digest structure that supports set intersection, union, and difference operations, can represent sets of arbitrary size with a fixed-size digest and has been applied in distributed cardinality estimation and set operation scenarios. OddSketch, on the other hand, constructs bit vector digests by performing bit-flipping encoding on set elements, enabling the estimation of the symmetric difference between two sets with extremely low space overhead.

[0003] While the aforementioned algorithms perform well in their respective applicable scenarios, existing data lake cleaning solutions still suffer from the following prominent problems. First, bandwidth bottlenecks during data migration severely restrict cleaning efficiency. The amount of raw data stored in data lakes typically reaches tens of terabytes or even petabytes, while the interconnect bandwidth between storage nodes and the host is limited. The time consumed in migrating the entire raw data to the host often far exceeds the time required for the actual deduplication calculation itself. Data transmission, rather than computational capability, becomes the performance bottleneck of the entire cleaning process. Second, existing solutions lack the ability to perform cleaning operations on-site at the storage end. Traditional storage device controllers are only responsible for physical read / write management of data and lack data processing logic. They cannot synchronously perform filtering or deduplication operations during data reading, resulting in a large amount of redundant data that will eventually be discarded still occupying valuable transmission bandwidth. Third, the communication overhead of cross-node near-duplicate detection is difficult to control. In a distributed storage architecture, near-duplicate records of the same entity may be distributed across different physical storage nodes. To discover these cross-node near-duplicate relationships, a large amount of raw data or complete signature information needs to be exchanged between nodes. The resulting communication overhead increases dramatically with the number of storage nodes. Fourth, most existing approximate deduplication algorithms are designed for a single similarity metric dimension and lack a collaborative mechanism for multi-level progressive screening. Relying solely on the judgment result of a single approximation algorithm to directly perform irreversible data deletion carries the risk of mistakenly deleting valid records. Simply chaining multiple algorithms together introduces redundant computational and communication overhead, making it difficult to strike a balance between detection accuracy and system efficiency. Fifth, in the resolution stage, existing solutions typically process near-duplicate record pairs independently, without considering the potentially chain-like or clustered structures of near-duplicate relationships, easily leading to inconsistent global resolution results. In summary, there is an urgent need for a data lake cleaning technology solution that can offload cleaning computation to the storage end, achieve cross-node near-duplicate detection with extremely low communication overhead, and possess multi-level progressive refinement verification and globally consistent resolution capabilities. Summary of the Invention

[0004] The main objective of this invention is to provide a near-memory processing unit and a data lake cleaning system for data cleaning. It enables the cleaning computation to be executed locally at the storage end, significantly reducing the bandwidth consumption of data migration. It completes the discovery of near duplicates across nodes with extremely low communication overhead. It achieves an effective balance between detection accuracy and system efficiency through a three-level progressive verification mechanism, and ensures the consistency and determinism of the global cleaning results through connected component-level resolution.

[0005] To solve the above problems, the technical solution of the present invention is implemented as follows:

[0006] A near-memory processing unit for data cleaning, integrated into the storage media controller layer, includes:

[0007] The signature generation module includes a multi-way hash calculation channel and an on-chip accumulation register group, which is used to generate SimHash fixed-length binary signatures for the original data records to be cleaned;

[0008] The segment partitioning and bucket mapping module includes a bucket mapping table storage area and a local candidate pair buffer. It is used to divide the SimHash fixed-length binary signature into multiple segments, establish a bucket mapping table with the segment identifier as the key, filter local near-duplicate candidate pairs in the same bucket according to the bucket mapping table, and extract the local segment identifier set for each segment position.

[0009] The Theta Sketch encoding module is used to construct the Theta Sketch structure from the local tape identifier set at each tape segment position and establish a reverse mapping relationship between the normalized hash value and the original tape identifier.

[0010] The Odd Sketch build module is used to construct Odd Sketch bit vectors from the set of attribute elements of the original data record;

[0011] The cleaning and elimination module includes a local cleaning result index table and a local space reclamation queue, which are used to register the storage address of retained records and reclaim the storage space of records to be eliminated.

[0012] Furthermore, the original data record is a line-based text data record. The process of generating a SimHash fixed-length binary signature by the signature generation module includes: splitting each original data record into multiple attribute fields according to column delimiters; performing sliding window segmentation on each attribute field according to a fixed byte length to obtain a sequence of feature fragments; sending each feature fragment into a multi-way hash calculation channel; each hash calculation channel using its own independent preset hash seed to perform hash operation on the feature fragment and output a binary intermediate vector of the same length as the target signature; performing bit-by-bit accumulation on all binary intermediate vectors generated by all feature fragments in the same original data record in the on-chip accumulation register group; after accumulation, performing sign determination on the accumulated value at each bit position; setting the position with a positive accumulated value to 1, and setting the position with a zero accumulated value to 0, thereby obtaining the SimHash fixed-length binary signature.

[0013] Furthermore, the working process of the segment partitioning and bucket mapping module includes: dividing the SimHash fixed-length binary signature into multiple continuous and non-overlapping segments along the bit direction. Each segment consists of a fixed number of continuous bits. The bit sequence of each segment is used as the segment identifier. A bucket mapping table is established on-chip for each segment position. The bucket mapping table uses the segment identifier as the key and the record identifier list as the value. The segment identifier of each original data record at each segment position is used as the index key. The corresponding record identifier is appended to the record identifier list of the corresponding segment identifier in the bucket mapping table. During the appending process, the number of existing record identifiers in the record identifier list is checked. When the number of existing record identifiers is greater than or equal to 1, the current record identifier is paired one by one with all existing record identifiers in the record identifier list to form local near-duplicate candidate pairs in the same bucket and written to the local candidate pair buffer.

[0014] Furthermore, the process of constructing the Theta Sketch structure by the Theta Sketch encoding module includes: applying a uniform hash transformation to each tag in the local tag set to map it to the open interval between 0 and 1 to obtain a normalized hash value; inserting all normalized hash values ​​one by one into an on-chip ordered array with a preset fixed capacity and maintaining ascending order; when the ordered array is full and a newly arrived normalized hash value is less than the last element of the array, replacing the last element with the new value and reordering; when a newly arrived normalized hash value is greater than or equal to the last element of the array, discarding it directly; after all tags have been processed, the ordered array is the local Theta Sketch structure; recording the largest normalized hash value in the ordered array as the local truncation threshold; and establishing a reverse mapping relationship from each normalized hash value to the original tag.

[0015] Furthermore, the process of constructing the Odd Sketch bit vector by the Odd Sketch building module includes: taking the complete byte content of each attribute field of the original data record as an attribute element, forming an attribute element set by combining all attribute elements of the same original data record, initializing a fixed-length all-zero bit vector on the chip, calculating a bit index value for each attribute element in the attribute element set through a preset hash function, locating the bit pointed to by the bit index value in the bit vector and performing a flip operation, that is, setting the bit to 1 if the current value of the bit is 0 and setting the bit to 0 if the current value of the bit is 1, and the bit vector obtained after processing all attribute elements is the Odd Sketch bit vector.

[0016] Data lake cleaning system, including:

[0017] Multiple storage nodes, each storage node has a near storage processing unit for data cleaning. The near storage processing unit is integrated into the storage medium controller layer and is used to generate SimHash fixed-length binary signatures for the original data records and perform band segmentation and in-band candidate pair screening, construct Theta Sketch structure and construct Odd Sketch bit vectors.

[0018] The coordinating node communicates with all storage nodes and is used to receive the ThetaSketch structure from each storage node according to the strip position and perform cross-node set intersection operations to form a global candidate pair set. Based on the Odd Sketch bit vector, it performs refined filtering and field-level precise comparison to confirm verified near-duplicate record pairs. Based on the verified near-duplicate record pairs, it constructs a near-duplicate graph and performs connectivity component-level resolution and data lake global metadata index update.

[0019] Furthermore, the cross-node set intersection operation process includes: the coordinating node takes the minimum value from the local truncation thresholds collected from all storage nodes for the same strip position as the global truncation threshold, retains only the normalized hash value entries in each local Theta Sketch structure that are strictly less than the global truncation threshold and merges them into a global Theta Sketch structure, filters out the normalized hash value entries that are simultaneously contained in two or more different storage nodes as cross-node co-occurrence entries, restores the cross-node co-occurrence entries to the corresponding strip identifiers through the reverse mapping relationship, locates the corresponding record identifier list through the strip bucket mapping table, pairs the record identifiers from different storage nodes to form cross-node near-duplicate candidate pairs, summarizes all cross-node near-duplicate candidate pairs and local same-bucket near-duplicate candidate pairs and removes duplicate pairs to form a global candidate pair set.

[0020] Furthermore, the process of refined filtering and field-level precise comparison includes: the coordinating node performs a bitwise XOR operation on the Odd Sketch bit vectors of two records in the same candidate pair to obtain the difference bit vector; the number of bits with a value of 1 in the difference bit vector is counted to obtain the parity difference count; candidate pairs with parity difference counts greater than or equal to the preset refined filtering threshold are removed; for candidate pairs with parity difference counts less than the preset refined filtering threshold, all attribute fields of the two records are compared column by column in the same column order to obtain the byte-level equality comparison; the proportion of attribute fields with completely identical content to the total number of attribute fields is counted as the field consistency rate; and candidate pairs with a field consistency rate greater than or equal to the preset precise confirmation threshold are confirmed as verified near-duplicate record pairs.

[0021] Furthermore, the process of connectivity component-level resolution includes: the coordinating node constructs a near-duplicate graph using the record identifiers of all verified near-duplicate record pairs as nodes and the pairing relationships as edges; it performs a connectivity component search on the near-duplicate graph to divide all nodes into several disjoint connected components, each of which constitutes a near-duplicate cluster; the coordinating node sends all record identifiers within each near-duplicate cluster to the corresponding storage node to request the data write timestamp of each record; within the same near-duplicate cluster, it compares the data write timestamps of all records and selects the record with the latest data write timestamp as the retained record, while the remaining records are marked as records to be eliminated. Each storage node's near-storage processing unit registers the storage address and record identifier of the retained records in the local cleaning result index table, writes the storage block addresses occupied by all records to be evicted into the local space reclamation queue, and performs erasure and release during the idle period by the storage media controller. After the erasure operation is completed, each storage node sends a local cleaning completion receipt to the coordinating node. After receiving the local cleaning completion receipts from all storage nodes, the coordinating node generates a global cleaning completion identifier and synchronously updates the number of valid records and the mapping relationship of storage addresses of the corresponding data partition in the global metadata index of the data lake according to the local cleaning result index table reported by each storage node.

[0022] Furthermore, each storage node is connected to the coordinating node through the inter-storage node interconnection channel. The original data records are line-based text data records. The near storage processing unit reads the original data records one by one from the read buffer of the storage medium controller and performs processing.

[0023] The present invention has the following beneficial effects: By integrating a near-memory processing unit for data cleaning into the storage medium controller layer, the present invention pushes the computational logic such as signature generation, strip partitioning, bucket mapping and filtering, Theta Sketch encoding and OddSketch construction down to the storage end where the data is located for execution. This allows the cleaning-related processing operations to be completed synchronously during the reading of the original data record from the storage medium. Only the cleaned effective data or a very small digest structure is transmitted to the host or coordinating node. This fundamentally avoids the practice of moving all the original data to the host for processing in traditional solutions, which significantly reduces the IO bandwidth consumption between the storage node and the host. As a result, the overall time of the cleaning process is no longer constrained by the data transmission bottleneck but mainly depends on the local processing speed of the storage end. This invention performs segmentation on SimHash fixed-length binary signatures and constructs Theta Sketch structures based on identifiers for cross-node set intersection operations. Each storage node only needs to upload a fixed-size Theta Sketch structure and its reverse mapping relationship to the coordinating node. The amount of communication data is negligible compared to the original data volume and does not increase linearly with the size of the data partition, thus maintaining controllable network overhead even in large-scale distributed deployments with a large number of storage nodes. Simultaneously, the segmentation mechanism utilizes the pigeonhole principle to ensure that two signatures with small Hamming distances are completely matched in at least one segment. This allows for the discovery of near-duplicate candidate pairs across nodes without transmitting the complete signature, balancing detection recall and communication efficiency. This invention constructs a three-level progressive verification mechanism: coarse screening with Theta Sketch, refined screening with Odd Sketch, and precise field-level comparison. Each level of screening gradually narrows the range of records requiring complete data transmission while eliminating a large number of false positive candidate pairs. This avoids the risk of false deletion caused by relying solely on a single approximation algorithm to directly determine data deletion, achieving an effective balance between detection accuracy and system efficiency. This invention constructs a near-duplicate graph using all verified near-duplicate records during the elimination phase and performs a connected component search. Records with transitive near-duplicate relationships are grouped into the same near-duplicate cluster for unified processing. Within each near-duplicate cluster, records are selected for retention based on their data write timestamps. This fundamentally solves the global inconsistency problem that may result from pairwise independent elimination, ensuring the determinism and reproducibility of the cleaning results at the cluster level. Furthermore, this invention schedules the storage space reclamation operation for records to be phased out during the idle period of the storage medium controller, avoiding I / O competition between the erase operation and normal business read / write operations. This minimizes the impact of the cleaning process on online business performance and is suitable for production environments with high service availability requirements. Attached Figure Description

[0024] Figure 1 This is a schematic diagram illustrating the SimHash fixed-length binary signature generation principle provided in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram illustrating the principle of Odd Sketch bit vector construction and refinement filtering provided in an embodiment of the present invention. Detailed Implementation

[0026] A near-memory processing unit for data cleaning, integrated into the storage media controller layer, includes:

[0027] The signature generation module includes a multi-way hash calculation channel and an on-chip accumulation register group, which is used to generate SimHash fixed-length binary signatures for the original data records to be cleaned;

[0028] The segment partitioning and bucket mapping module includes a bucket mapping table storage area and a local candidate pair buffer. It is used to divide the SimHash fixed-length binary signature into multiple segments, establish a bucket mapping table with the segment identifier as the key, filter local near-duplicate candidate pairs in the same bucket according to the bucket mapping table, and extract the local segment identifier set for each segment position.

[0029] The Theta Sketch encoding module is used to construct the Theta Sketch structure from the local tape identifier set at each tape segment position and establish a reverse mapping relationship between the normalized hash value and the original tape identifier.

[0030] The Odd Sketch build module is used to construct Odd Sketch bit vectors from the set of attribute elements of the original data record;

[0031] The cleaning and elimination module includes a local cleaning result index table and a local space reclamation queue, which are used to register the storage address of retained records and reclaim the storage space of records to be eliminated.

[0032] In data lake scenarios, the raw data records accumulated in storage clusters often reach billions or even higher, including a large number of near-duplicate records introduced by processes such as multi-source collection, repeated storage, and incremental appending. The traditional approach is to move all data to the host-side compute nodes for cleaning, a process constrained by the I / O bandwidth between the storage nodes and the host. Assuming an average of 512 bytes per raw data record, the data movement volume for 1 billion records would be approximately 476 gigabytes. If the storage interconnect bandwidth is 12.8 gigabytes per second, the data transmission alone would take about 37 seconds, while the actual cleaning computation time is likely much less. In other words, the bottleneck lies in the data movement, not the computation itself. Therefore, this embodiment decentralizes the lightweight data cleaning logic to the storage media controller layer, allowing filtering and deduplication operations to be completed simultaneously with data reading. Only the cleaned, valid data is transmitted to the host, thereby reducing the I / O bandwidth requirement to a fraction of the original data volume.

[0033] The following section provides a detailed explanation of the complete implementation process of distributed near-duplicate resolution, using a specific near-memory processing unit architecture as an example.

[0034] The process of generating a SimHash fixed-length binary signature in the near-memory processing unit is as follows. The near-memory processing unit resides at the storage media controller layer and possesses independent on-chip logical resources, including multiple hash calculation channels and an on-chip accumulation register set. When the storage media controller responds to a data read request, the original data record first arrives at the read buffer. The near-memory processing unit extracts the original data records to be cleaned one by one from the read buffer. In this embodiment, the original data records are line-based text data records, each consisting of multiple attribute fields separated by column delimiters, which can be tabs, commas, or vertical bars. In a typical configuration, each original data record contains 8 attribute fields, with each attribute field having an average length of 48 bytes.

[0035] For each original data record, the near-storage processing unit splits it into multiple attribute fields according to column delimiters. Then, a sliding window is performed on each attribute field with a fixed byte length. Here, a fixed byte length of 4 bytes and a sliding step of 1 byte are chosen. The reason for choosing a 4-byte window is that 4 bytes precisely cover one complete UTF-8 encoded Chinese character or two ASCII characters, capturing sufficiently fine-grained local features without generating too many meaningless fragments due to an excessively small window. If the attribute field length is 48 bytes, the sliding window segmentation yields 45 feature fragments. In optional implementations, the fixed byte length can be set to 3 bytes, 5 bytes, or 6 bytes, depending on the character encoding type of the data record and the average length of the attribute fields.

[0036] Each feature fragment is fed into a multi-channel hash calculation process within the near-memory processing unit. This embodiment uses 64 hash calculation channels to generate a 64-bit SimHash fixed-length binary signature. Each hash calculation channel uses its own independent preset hash seed, which is written into the on-chip configuration register by the host during the near-memory processing unit initialization phase and remains unchanged throughout the entire cleaning cycle. Each hash calculation channel performs a hash operation on the input feature fragment, outputting a 64-bit binary intermediate vector equal in length to the target signature. The hash operation can be implemented using the MurmurHash3 algorithm, which has low computational complexity and good uniformity, making it suitable for implementation as a combinational circuit in on-chip logic. In optional implementations, the hash operation can also use the xxHash or CityHash algorithms.

[0037] In the binary intermediate vector output by each hash calculation channel, positions with a bit value of 1 are considered positive contributions, and positions with a bit value of 0 are considered negative contributions. For all binary intermediate vectors generated from all feature segments in the same original data record, bit-by-bit accumulation is performed in the on-chip accumulation register set. The on-chip accumulation register set contains 64 independent accumulation registers, each corresponding to one bit position of the SimHash fixed-length binary signature. The specific accumulation rule is as follows: for the... 1 bit position, of which The value ranges from 0 to 63. If a certain binary intermediate vector is at the 63rd position... If the value of a position is 1, the value of the corresponding accumulator register is incremented by 1; if the value of that position is 0, the value of the corresponding accumulator register is decremented by 1. This is the bit position index. After all feature fragments of all attribute fields of the original data record have been processed, a sign determination is performed on the accumulated value of each bit position in the on-chip accumulator register group: the position with a positive accumulated value is set to 1, and the position with a negative accumulated value is set to 0. This generates the SimHash fixed-length binary signature of the original data record. Intuitively, this process means that if a majority of feature fragments vote 1 for a certain bit, then the final signature for that bit is determined to be 1, and vice versa. Two records with similar content tend to have consistent voting results because they share a large number of identical feature fragments, resulting in a smaller Hamming distance between the final SimHash fixed-length binary signatures.

[0038] In an optional implementation, the length of the SimHash fixed-length binary signature can be set to 128 bits or 256 bits. A longer signature provides stronger differentiation between different records, but also increases on-chip storage overhead and comparison operation overhead. For scenarios with no more than 16 attribute fields and a total record length of no more than 1024 bytes, a 64-bit signature is usually sufficient to meet the differentiation requirements.

[0039] After generating the SimHash fixed-length binary signature, the near-memory processing unit performs segmentation on it. The SimHash fixed-length binary signature is divided into multiple consecutive and non-overlapping segments along the bit direction. In this embodiment, the 64-bit signature is divided into 4 segments, each consisting of 16 consecutive bits. The bit sequence of each segment is used as the segment identifier. Taking the first segment as an example, if the value of bits 0 to 15 of the SimHash fixed-length binary signature is 0110100110001011, then the segment identifier for the first segment is 0110100110001011.

[0040] refer to Figure 1 ,like Figure 1As shown, the near-memory processing unit performs signature generation on the raw data records to be cleaned, and the complete data processing flow is divided into six stages from top to bottom. In the first stage, the near-memory processing unit extracts one raw data record from the read buffer of the storage medium controller. This raw data record is a row-oriented text data record, consisting of multiple attribute fields connected by column separators. The example in the figure includes eight attribute fields, from field 1 to field 8. In the second stage, the near-memory processing unit splits the raw data record according to the column separators to obtain each independent attribute field. Then, it performs sliding window segmentation on each attribute field according to a fixed byte length. The figure uses a hexadecimal byte sequence to illustrate the working method of the sliding window. The three dashed rectangles represent the window sliding one byte to the right in sequence, each time extracting a continuous segment of 4 bytes as a feature segment, thereby transforming the attribute fields into a sequence of feature segments. In the third stage, all feature fragments are fed into multiple hash calculation channels within the near-memory processing unit. The diagram shows 64 parallel hash calculation channels, from channel 0 to channel 63. Each channel has its own independent preset hash seed; for example, channel 0's seed is 0x1A3F, channel 1's seed is 0x7B02, and channel 63's seed is 0x32AF. Each hash calculation channel performs a hash operation on the input feature fragments, outputting a 64-bit binary intermediate vector of the same length as the target signature. The diagram shows the bit sequence fragments of the intermediate vector below each channel, such as channel 0 outputting 10110010, and channel 1 outputting 01101001. In the fourth stage, all binary intermediate vectors generated from all feature fragments in the same original data record are fed into an on-chip accumulation register group for bit-by-bit accumulation. The diagram shows 64 accumulation registers, corresponding to bits 0 to 63 of the signature. Each register stores the accumulated result of all intermediate vectors at that bit position. The accumulation rule is that when the value of the intermediate vector at a certain position is 1, the corresponding register is incremented by 1; when the value is 0, the corresponding register is decremented by 1. As shown in the diagram, the accumulated values ​​of each register can be positive or negative. For example, the accumulated value of bit 0 is +12, bit 1 is -5, bit 2 is +8, and bit 7 is +15, etc. In the fifth stage, a sign determination is performed on the accumulated value at each bit position in the accumulation register group. The rule is that positions with positive accumulated values ​​are set to 1, and positions with 0 accumulated values ​​are set to 0. After the sign determination, the 64 accumulated values ​​are converted into 64 binary bits, which are concatenated to form a complete SimHash fixed-length binary signature. The bottom of the diagram shows the generated 64-bit signature sequence, where cells with darker backgrounds represent bit values ​​of 1, and cells with blank backgrounds represent bit values ​​of 0.In the sixth stage, the segmentation structure of the signature is illustrated in square brackets. The 64-bit signature is divided into four consecutive and non-overlapping segments: segment 1 (bits 0 to 15), segment 2 (bits 16 to 31), segment 3 (bits 32 to 47), and segment 4 (bits 48 to 63). Each segment contains 16 consecutive bits. This segmentation is fundamental for building the bucket mapping table and performing cross-node Theta Sketch set intersection operations. Two records with similar content vote on most bits due to sharing a large number of feature fragments, resulting in a small Hamming distance between signatures. At least one segment among the four segments has a completely matching bit sequence, thus transforming global pairwise comparisons into local comparisons within the buckets.

[0041] The design intent of the strip partitioning is that there is a very high probability that at least one of the segments in a SimHash fixed-length binary signature with a small overall Hamming distance will have the same bit sequence across multiple segments. Taking a 64-bit signature divided into 4 segments as an example, if the Hamming distance between two signatures is no more than 3 bits, then at least one segment will have 16 bits that are completely identical. This is because the 3 different bits are distributed across the 4 segments, and according to the pigeonhole principle, at least one segment will not contain any different bits. Utilizing this property, the global pairwise comparison problem can be transformed into a local comparison problem within the strip buckets, significantly reducing computational complexity. In an optional implementation, the number of segments can be set to 8, with each segment containing 8 bits. This configuration can detect nearly identical pairs with a Hamming distance of no more than 7, but it generates more candidate pairs, increasing the overhead of subsequent verification.

[0042] A tape bucket mapping table is created on-chip for each tape segment location. The tape bucket mapping table uses the tape identifier as the key and the record identifier list as the value. The tape bucket mapping table is stored in the on-chip SRAM near the storage processing unit. In actual implementation, considering the limited capacity of the on-chip SRAM, typically 256 kilobytes to 1 megabyte, the tape bucket mapping table is implemented using an open-address hash table structure. Each slot stores one tape identifier and its corresponding record identifier list head pointer. The record identifier list itself is stored as a linked list in the dynamically allocated area of ​​the on-chip SRAM. If the number of records in a data partition exceeds the capacity of the on-chip SRAM, the earlier record identifier list overflows into the off-chip DRAM buffer managed by the storage media controller.

[0043] When processing each raw data record, its band identifier at each band segment position is used as the index key, and the corresponding record identifier is appended to the record identifier list of the corresponding band identifier in the band bucket mapping table. During the appending process, the number of existing record identifiers in the record identifier list is checked. When the number of existing record identifiers is greater than or equal to 1, it means that at least one record has previously generated the exact same band identifier at the same band segment position. The same band identifier means that the SimHash fixed-length binary signatures of these records are completely identical in the 16 bits of that band segment, so they are potential near-duplicate records. At this time, the current record identifier is paired one by one with all existing record identifiers in the record identifier list to form local near-duplicate candidate pairs in the same bucket, and written to the local candidate pair buffer. The local candidate pair buffer is also located in the on-chip SRAM and adopts a circular queue structure. When the buffer is full, the existing contents are flushed to the off-chip DRAM in batches.

[0044] After all the original data records have been processed, the near-storage processing unit extracts all the band identifiers that have appeared under each band position from the all-band-bucket mapping table of all band positions, forming a local band identifier set for each band position. Taking 4 band positions as an example, this forms 4 local band identifier sets. The number of elements in each local band identifier set depends on the diversity of the data. If the data partition contains 1 million records, the number of elements in the local band identifier set of a single band position is usually between 100,000 and 800,000.

[0045] Next, a Theta Sketch structure is constructed for the local band identifier set at each band position. The Theta Sketch structure is a probabilistic data digest structure that supports set intersection, union, and difference operations. Its core idea is to apply a uniform hash transformation to the elements of the set and retain only the samples with the smallest hash values, thereby representing a set of arbitrary size with a digest of a fixed size.

[0046] The construction process is as follows: A uniform hash transformation is applied to each identifier in the local identifier set, mapping it to an open interval between 0 and 1 to obtain a normalized hash value. The specific implementation of the uniform hash transformation is as follows: First, a 64-bit MurmurHash3 operation is performed on the bit sequence of the identifier to obtain an unsigned integer hash value. ,in For a 64-bit unsigned integer, then... Divide by the maximum value of a 64-bit unsigned integer Obtain the normalized hash value ,in This represents a normalized floating-point number falling within the open interval between 0 and 1. Since the output of MurmurHash3 is approximately uniformly distributed... The distribution is also approximately uniform.

[0047] One fixed capacity is allocated on the chip. A sorted array, in which This refers to the sampling capacity parameter of the Theta Sketch structure. In this embodiment... Take 4096. The value of directly affects the precision of set operations: The larger the value, the smaller the estimation error, but the greater the communication and storage overhead. For data lake scenarios with collections in the millions, When the set intersection value is 4096, the relative error is approximately 1.5%, which is sufficient to meet the accuracy requirements for near-repeating detection. In an optional implementation, You can choose 1024 or 8192.

[0048] Normalized hash values ​​are inserted one by one into the sorted array, always maintaining ascending order. When the sorted array is full and a newly arrived normalized hash value is less than the last element of the array, the last element is replaced with the new value, and binary insertion is used to maintain ascending order. Newly arrived normalized hash values ​​greater than or equal to the last element of the array are discarded. After all labeled processing is complete, the sorted array becomes the local Theta Sketch structure. The largest normalized hash value in the sorted array is recorded as the local truncation threshold. ,in This represents the truncation boundary of the Theta Sketch at the current storage node's current band segment location. Simultaneously, a reverse mapping relationship is established between each normalized hash value and the original band identifier. This reverse mapping relationship is stored in on-chip SRAM as key-value pairs with the normalized hash value as the key and the original band identifier as the value. The purpose of this reverse mapping relationship is that after the coordinating node finds cross-node co-occurrence entries through set intersection, it needs to restore the normalized hash value of the co-occurrence entries back to the original band identifier, and then locate the specific record identifier through the band bucket mapping table, thus completing the backtracking from the probabilistic digest to the deterministic record identifier.

[0049] Each storage node transmits its local Theta Sketch structure for all strip locations, along with the corresponding local truncation thresholds and inverse mappings, to the coordinating node via the inter-storage node interconnect channel. This is based on four strip locations... Taking 4096 as an example, each local Theta Sketch structure contains 4096 normalized hash values, and each normalized hash value occupies 8 bytes. Therefore, the amount of Theta Sketch data uploaded by a single storage node is 4 segments multiplied by 4096 multiplied by 8 bytes, which equals 131072 bytes, or 128 kilobytes, which is much smaller than the original data volume.

[0050] After receiving the local Theta Sketch structure reported by all storage nodes, the coordinating node processes it according to the stripe position. For the same stripe position, the minimum value among the local truncation thresholds collected from all storage nodes is taken as the global truncation threshold. ,in This is the minimum of all local truncation thresholds. The reason for choosing the minimum is that Theta Sketch's set intersection operation requires all participating Sketches to be compared under the same truncation boundary; using the strictest truncation threshold ensures that all retained entries are valid on each node. It iterates through all normalized hash value entries in the local Theta Sketch structure reported by each storage node, retaining only those strictly below the global truncation threshold. The entries are merged into a global ThetaSketch structure.

[0051] In the global Theta Sketch structure, entries with normalized hash values ​​simultaneously contained in two or more different storage nodes are selected and marked as cross-node co-occurrence entries. Cross-node co-occurrence entries mean that there are records with completely identical identifiers on different storage nodes. Identical identifiers mean that the SimHash fixed-length binary signatures of the corresponding records are completely identical in the bits of that segment. Therefore, these records are likely to be near-duplicate records across nodes. Cross-node co-occurrence entries are restored to their corresponding identifiers through a reverse mapping relationship. Then, the identifiers are located to the corresponding record identifier lists through the bucket mapping table of each storage node. Record identifiers from different storage nodes are paired to form cross-node near-duplicate candidate pairs. All cross-node near-duplicate candidate pairs and the local bucket near-duplicate candidate pairs generated by each storage node are aggregated, and duplicate pairings are removed to form a global candidate pair set.

[0052] Up to this point, the candidate pairs in the global candidate pair set are still based on the coarse screening results at the band level, which inevitably contain some false positive pairs, meaning that two records only happen to match on a certain band, but their overall content is quite different. Therefore, further refined screening is needed to eliminate false positives.

[0053] The coordinating node sends back the record identifiers for each candidate pair in the global candidate pair set to its respective storage node. Each storage node's near-memory processing unit retrieves the corresponding original data record from the storage medium based on the received record identifier. For each attribute field of the original data record, the complete byte content is taken as one attribute element, and all attribute elements of the same original data record are combined to form an attribute element set. For example, a record containing 8 attribute fields will have an attribute element set containing 8 attribute elements.

[0054] Construct Odd Sketch bit vectors for a set of attribute elements. Odd Sketch is a set digest structure based on bit flipping. Its core idea is: for each element in the set, a position in the bit vector is determined by a hash function and the bit value at that position is flipped. If the contents of two sets are exactly the same, the bit vectors obtained after the same flipping sequence will also be exactly the same; conversely, the greater the difference between the two sets, the more differences there are in the flipping operations, and the greater the Hamming distance of the final bit vectors.

[0055] The specific construction process is as follows: A fixed-length all-zero bit vector is initialized on-chip. In this embodiment, the bit vector length is set to 256 bits. For each attribute element in the attribute element set, a bit index value is calculated using a preset hash function. The bit index value ranges from 0 to 255. The preset hash function can also use the MurmurHash3 algorithm; the lower 8 bits of its output are modulo 256 to obtain the bit index value. The bit pointed to by the bit index value is located in the bit vector, and a flip operation is performed: if the current value of the bit is 0, it is set to 1, and if the current value of the bit is 1, it is set to 0. After processing all attribute elements, the resulting bit vector is the Odd Sketch bit vector.

[0056] The design intent of the flip operation needs special explanation here. Unlike Bloom filters, which only set bits to 1, OddSketch performs a flip operation. This means that if two different attribute elements happen to map to the same position, the first element flips the bit from 0 to 1, and the second element flips the bit back from 1 to 0. This mechanism makes OddSketch sensitive to the parity of the number of elements in the set, enabling it to detect subtle differences between sets. In optional implementations, the bit vector length can be set to 512 bits or 1024 bits. The longer the bit vector, the lower the probability of hash collisions between different attribute elements and the higher the detection accuracy, but the transmission overhead also increases accordingly. For typical scenarios with 8 to 32 attribute fields, a bit vector length of 256 bits achieves a good balance between accuracy and overhead.

[0057] Each storage node transmits the generated Odd Sketch bit vector along with the corresponding record identifier to the coordinating node. Each Odd Sketch bit vector occupies only 32 bytes, significantly saving bandwidth compared to transmitting the complete original data record. The coordinating node performs a bitwise XOR operation on the Odd Sketch bit vectors of two records from different storage nodes in the same candidate pair to obtain a difference bit vector, and counts the number of bits with a value of 1 in the difference bit vector to obtain a parity difference count. If the attribute element sets of the two records are completely identical, then the Odd Sketch bit vectors are also completely identical, and the parity difference count is 0. The greater the difference in the attribute element sets, the larger the parity difference count.

[0058] refer to Figure 2 ,like Figure 2As shown, the upper part of the diagram is divided into two symmetrical regions, illustrating the process of constructing Odd Sketch bit vectors for each of the two records in a candidate pair. The left side represents record A located at storage node 1, and the right side represents record B located at storage node 2. For record A, the complete byte content of all its attribute fields is first extracted into a set of attribute elements. The diagram shows 8 attribute elements, with the value of the 5th attribute element being 35. For record B, 8 attribute elements are also extracted, with the value of the 5th attribute element being 36, and the remaining 7 attribute elements being identical to those of record A. This means that there is only one attribute field difference between record A and record B. Next, an Odd Sketch bit vector is constructed for the attribute element set of each record. The construction process first initializes a fixed-length 256-bit all-zero bit vector on-chip. Then, for each attribute element in the attribute element set, a bit index value is calculated using a preset hash function. The bit pointed to by this index value is located in the bit vector, and a flip operation is performed: if the current value of the bit is 0, it is set to 1, and if the current value of the bit is 1, it is set to 0. The diagram illustrates the hash mapping results and flipping process of each attribute element in a step-by-step manner. Taking record A as an example, the attribute element "Zhang San" is hashed to bit 42, which flips from 0 to 1; the attribute element "Beijing" is mapped to bit 117, which flips from 0 to 1; the attribute element "35" is mapped to bit 42, and bit 42, which had previously been flipped to 1, is flipped back to 0; the attribute element "A12" is mapped to bit 203, which flips from 0 to 1. For record B, the hash mapping results of most attribute elements are exactly the same as those of record A. The key difference is that the 5th attribute element of record B is "36," which hashes to bit 89 instead of bit 42, so bit 89 flips from 0 to 1. After all attribute elements have been processed, the diagram shows the final Odd Sketch bit vectors of record A and record B on the left and right sides, respectively. Only a few bits in the bit vectors have a value of 1, while the rest are 0. A comparison reveals that record A has a 0 at bit 42 while record B has a 1 at bit 42, and record A has a 0 at bit 89 while record B has a 1 at bit 89. These two positions constitute the difference between the two bit vectors. The lower half of the diagram illustrates the process by which the coordinating node refines and filters the two Odd Sketch bit vectors. After receiving the Odd Sketch bit vectors of two records from different storage nodes, the coordinating node performs a bitwise XOR operation on the two bit vectors to obtain the difference bit vector. The bits with a value of 1 in the difference bit vector correspond to the positions in the two Odd Sketch bit vectors where the values ​​differ. In the diagram, bits 42 and 89 in the difference bit vector are marked with a darker background, while the remaining positions are 0. The coordinating node counts the number of bits with a value of 1 in the difference bit vector to obtain the parity difference count.The decision box in the figure shows that the odd-even difference count is 2, which is less than the preset refined screening threshold of 20. Therefore, the candidate pair passes the refined screening and enters the field-level precise comparison stage for final confirmation.

[0059] The parity difference count is compared with a preset refinement screening threshold. In this embodiment, the preset refinement screening threshold is set to 20. When the parity difference count is greater than or equal to the preset refinement screening threshold, it indicates that there is a significant difference in the attribute element sets of the two records, and it is judged as a false alarm and removed from the global candidate pair set. When the parity difference count is less than the preset refinement screening threshold, it indicates that the attribute element sets of the two records have a high degree of similarity, and they are retained for the next stage of field-level precise comparison. In an optional implementation, the preset refinement screening threshold can be set to an integer between 10 and 40, depending on the data characteristics and deduplication accuracy requirements.

[0060] It is important to note that, as a probabilistic data structure, Odd Sketch inherently has a certain false positive rate. Therefore, it is not advisable to directly perform irreversible data deletion operations based on the judgment results of Odd Sketch. In this embodiment, the role of Odd Sketch is to quickly eliminate obviously dissimilar candidate pairs, thereby reducing the computational load of subsequent precise field-level comparisons.

[0061] Candidate pairs refined through Odd Sketch enter the field-level precise comparison stage. The coordinating node requests all attribute field contents of the corresponding original data records from the storage nodes involved in the candidate pair. Since the number of remaining candidate pairs has been significantly reduced after Odd Sketch refinement, the bandwidth overhead of transmitting complete attribute field contents is controllable. The attribute fields of the two records are compared column-by-column in the same column order, byte-level equality. The proportion of attribute fields with completely identical content to the total number of attribute fields is used as the field consistency rate. Taking a record with 8 attribute fields as an example, if 6 of the attribute fields are completely identical, the field consistency rate is 0.75. Candidate pairs with a field consistency rate greater than or equal to a preset precise confirmation threshold are confirmed as verified near-duplicate record pairs. In this embodiment, the preset precise confirmation threshold is set to 0.8, meaning that at least 80% of the attribute field contents are completely identical before being confirmed as near-duplicate. In an optional implementation, the preset precise confirmation threshold can be set to a value between 0.7 and 0.95, depending on the strictness of deduplication requirements of the business scenario. Candidate pairs with a field consistency rate less than the preset precise confirmation threshold are removed.

[0062] After the above multi-level screening, the verified near-duplicate record pairs enter the final elimination stage. The goal of elimination is to retain one representative record in each near-duplicate record pair, discard the remaining redundant records, and reclaim storage space.

[0063] However, near-duplicate relationships are not simply pairwise independent relationships, but may exhibit chain-like or cluster-like structures. For example, record A and record B form a verified near-duplicate record pair, and record B and record C also form a verified near-duplicate record pair, but record A and record C are not directly paired. If processed by eliminating each pair independently, globally inconsistent results may occur. For example, in the AB pair, A is retained while B is eliminated, while in the BC pair, B is retained while C is eliminated, resulting in B being both eliminated and retained. To avoid this problem, the coordinating node constructs a near-duplicate graph using the record identifiers of all verified near-duplicate record pairs as nodes and the pairing relationships as edges. A connected component search is performed on the near-duplicate graph, dividing all nodes into several disjoint connected components. Each connected component constitutes a near-duplicate cluster. The connected component search can be implemented using breadth-first traversal or a disjoint-set data structure (union-find). In optional implementations, when the number of verified near-duplicate record pairs does not exceed 100,000, the disjoint-set data structure is used, with a time complexity close to linear; when the number is larger, a block-parallel connected component merging strategy can be used.

[0064] The coordinating node distributes all record identifiers within each near-duplication cluster to the corresponding storage node, requesting the data write timestamp for each record. Within the same near-duplication cluster, the data write timestamps of all records are compared, and the record with the latest data write timestamp is selected as the retained record. The reason for selecting the latest record is that in the incremental update mode of the data lake, subsequent records of the same entity often carry more complete or newer information; retaining the latest version maximizes the timeliness and integrity of the data. In an optional implementation, the retention strategy can also be set to select the record with the most non-null attribute fields, or the record with the highest data source priority. The remaining records are marked as pending elimination records.

[0065] Each storage node's near-storage processing unit retains the storage address and record identifier of the records and registers them in a local cleaning result index table. This local cleaning result index table records the physical storage locations of all records that remain valid after cleaning on the local storage node. Subsequent data lake query engines can directly locate valid data through this index table without accessing obsolete records. The storage block addresses occupied by all records awaiting obsolescence are written to a local space reclamation queue. This queue uses a first-in, first-out (FIFO) structure; the storage media controller retrieves the storage block address from the head of the queue during idle periods and performs an erase and release operation. Scheduling space reclamation during idle periods avoids I / O contention between erase operations and normal data read / write operations. For flash-based storage media, erase operations are much more time-consuming than read / write operations; performing erase synchronously on the critical path of data cleaning would significantly extend the overall cleaning time.

[0066] After each storage node completes the removal of all near-duplicate clusters on its local node, it sends a local cleaning completion receipt to the coordinating node. Upon receiving all local cleaning completion receipts from the storage nodes, the coordinating node generates a global cleaning completion identifier and synchronously updates the mapping relationship between the number of valid records and storage addresses for the corresponding data partition in the data lake's global metadata index based on the local cleaning result index table reported by each storage node. Once the global metadata index is updated, the data lake can provide query services based on the cleaned data.

[0067] In summary, the above implementation process decomposes distributed near-duplicate resolution into six stages: signature generation, segment partitioning and bucket mapping, Theta Sketch cross-node intersection discovery, Odd Sketch refinement and filtering, field-level precise confirmation, and connectivity component-level resolution. Each stage is completed during the data flow, minimizing data movement between storage nodes and the host. In an experimental configuration with 8 storage nodes, each storing 100 million original data records, compared with the traditional full data movement and cleaning method, the above near-duplicate storage processing method reduces IO bandwidth consumption to approximately 12% of the original solution and cleaning time to approximately 28%. The recall rate of near-duplicate record pairs has been verified to be no less than 96.5%, and the false deletion rate is no more than 0.3%.

[0068] Data lake cleaning system, including:

[0069] Multiple storage nodes, each storage node has a near storage processing unit for data cleaning. The near storage processing unit is integrated into the storage medium controller layer and is used to generate SimHash fixed-length binary signatures for the original data records and perform band segmentation and in-band candidate pair screening, construct Theta Sketch structure and construct Odd Sketch bit vectors.

[0070] The coordinating node communicates with all storage nodes and is used to receive the ThetaSketch structure from each storage node according to the strip position and perform cross-node set intersection operations to form a global candidate pair set. Based on the Odd Sketch bit vector, it performs refined filtering and field-level precise comparison to confirm verified near-duplicate record pairs. Based on the verified near-duplicate record pairs, it constructs a near-duplicate graph and performs connectivity component-level resolution and data lake global metadata index update.

[0071] In practical operation, data lakes often employ a distributed storage architecture, with raw data records distributed across multiple physical storage nodes according to hash partitioning or range partitioning strategies. This distributed layout presents a core challenge: near-duplicate records of the same entity may be distributed across different storage nodes, and each storage node can only perceive its local data and cannot independently detect near-duplicate relationships across nodes. If local deduplication is performed only within each storage node, records that only have duplicate relationships between nodes will be systematically missed, resulting in incomplete cleaning. To address this, this embodiment proposes a data lake cleaning system that deploys near-duplicate processing units on each storage node and introduces a coordinating node to perform cross-node information aggregation and global resolution, achieving complete distributed near-duplicate detection and resolution.

[0072] The overall topology of the data lake scrubbing system in this embodiment is as follows. The system includes multiple storage nodes and one coordinating node. In a typical deployment configuration, there are eight storage nodes, each managing a data partition with a capacity of 500 gigabytes to 2 terabytes, corresponding to approximately 100 million to 4 billion raw data records. The coordinating node communicates with all storage nodes through an inter-node interconnect channel. The inter-node interconnect channel can be implemented using a remote direct memory access network, with a single-link bandwidth of 25 gigabits per second or 100 gigabits per second. In optional implementations, the inter-node interconnect channel can also use an Ethernet or InfiniBand network, depending on the data center infrastructure configuration. The coordinating node can be a standalone physical server, or it can be one of the storage nodes in the storage cluster. The latter can save additional hardware costs but will have a certain impact on the storage processing performance of that node.

[0073] Each storage node contains a near-memory processing unit for data cleaning, integrated within the storage media controller layer. The storage media controller is the core chip inside the storage device that manages flash memory chips or disk media, directly controlling physical data read and write operations. The reason for integrating the near-memory processing unit at this layer instead of the host-side software stack is that data read from the storage medium first passes through the storage media controller. If the cleaning-related calculations are completed here, there is no need to move the entire original data to the host via the storage interconnect channel, thus physically eliminating the data migration bottleneck. The on-chip logic resource area of ​​the near-memory processing unit accounts for approximately 8% to 15% of the total logic area of ​​the storage media controller, and the on-chip SRAM occupies approximately 256 kilobytes to 1 megabyte. This resource overhead is within the design margins of modern storage controller chips.

[0074] The data lake cleaning system's workflow is divided into four stages according to time sequence: local processing, cross-node aggregation, refinement and verification, and global resolution. There are strict data dependencies between each stage; the output of one stage becomes the input of the next.

[0075] During the local processing phase, the coordinating node broadcasts a cleaning start command to all storage nodes. This command carries parameters such as the target data partition identifier, the bit length configuration of the SimHash fixed-length binary signature, and the number of segments. Upon receiving the cleaning start command, each storage node's near-storage processing unit reads the original data records within the target data partition one by one from the read buffer of the storage media controller. It then sequentially completes the generation of the SimHash fixed-length binary signature, the segment partitioning and establishment of the bucket mapping table, the screening of local near-duplicate candidate pairs within the same bucket, and the extraction of the local band identifier set for each segment location. The specific technical details of these operations have been described in detail in the near-storage processing unit's implementation. Here, we focus on the output dataset generated by each storage node at the end of the local processing phase. Each storage node generates three types of output after the local processing phase: the first type is the local near-duplicate candidate pair set within the same bucket, stored in the local candidate pair buffer; the second type is the local Theta Sketch structure for each segment location along with the local truncation threshold and reverse mapping relationship; and the third type is the bucket mapping table, retained on-chip for subsequent querying.

[0076] The cross-node aggregation phase is led by the coordinating node. Each storage node transmits its Type 2 output—the local Theta Sketch structure of all segment locations, along with the corresponding local truncation thresholds and inverse mappings—to the coordinating node via the inter-node interconnection channel. The transmission employs an asynchronous, non-blocking method: each storage node initiates data transmission immediately after completing its local processing, without waiting for other nodes to finish. The coordinating node maintains a reception status table, recording the reported status of each storage node. Once data has arrived from all storage nodes, the cross-node aggregation operation phase begins.

[0077] The scale of data transmission directly affects the system's network overhead. This is illustrated by 8 storage nodes, 4 strip locations, and a Theta Sketch sampling capacity. Taking 4096 as an example, each Theta Sketch structure contains Each storage node has a normalized hash value, each occupying 8 bytes. In the reverse mapping relationship, each mapping record contains an 8-byte normalized hash value and a 2-byte identifier, totaling 10 bytes. The total data uploaded by a single storage node is approximately 4 segments multiplied by 4096 entries multiplied by 18 bytes, equaling 294912 bytes, or approximately 288 kilobytes. The eight storage nodes collectively upload approximately 2.25 megabytes. Compared to the hundreds of gigabytes of raw data on the eight storage nodes, this communication overhead is negligible. In an optional implementation, when the number of storage nodes reaches 64 or more, a tree-structured aggregation topology can be used instead of a star-structured direct-connection topology. That is, every eight storage nodes first converge to an intermediate node for local merging, and then the intermediate node reports the merge result to the coordinating node, thereby avoiding the coordinating node becoming a network bandwidth bottleneck.

[0078] After receiving the local Theta Sketch structure from all storage nodes, the coordinating node processes it according to the stripe position. For each stripe position, the coordinating node performs the following cross-node set intersection operation. First, it selects the minimum value from the local truncation thresholds reported by all storage nodes as the global truncation threshold. ,in This is a scalar value representing the globally uniform ThetaSketch truncation boundary. The minimum value is chosen because the size of the local tag set varies across storage nodes, resulting in different ThetaSketch sampling densities. Using the minimum truncation threshold ensures fair comparisons under a uniform sampling density. Without a uniform truncation threshold, nodes with larger sets will have lower truncation thresholds and denser sampling, while nodes with smaller sets will have higher truncation thresholds and sparser sampling. Direct comparison between the two would introduce systematic bias.

[0079] Then, iterate through all normalized hash value entries in the local Theta Sketch structure reported by each storage node, retaining only those strictly below the global truncation threshold. The entries are merged and arranged in ascending order of normalized hash value to form a global Theta Sketch structure. During the merging process, the coordinating node records the source storage node identifier for each entry. The global Theta Sketch structure is scanned, and entries with normalized hash values ​​contained in two or more different storage nodes are selected and marked as cross-node co-occurrence entries.

[0080] The coordinating node restores each cross-node co-occurrence entry to its corresponding tag identifier using a reverse mapping relationship, and then sends tag identifier query requests to each relevant storage node. Each storage node locates the tag identifier to the corresponding record identifier list according to the tag bucket mapping table and returns it to the coordinating node. The coordinating node pairs record identifiers from different storage nodes to form cross-node near-duplicate candidate pairs. A detail needs to be addressed here: the same tag identifier may correspond to multiple records within a single storage node; therefore, the tag identifier query returns a list of record identifiers rather than a single record identifier. During pairing, the coordinating node performs a cross-node Cartesian product, that is, it pairs each record identifier in the record identifier list from storage node A with each record identifier in the record identifier list from storage node B one by one. To control the expansion of the number of candidate pairs, if the number of record identifiers corresponding to a single tag identifier on a certain storage node exceeds the preset maximum list length (set to 256 in this embodiment), only the first 256 are used for pairing. In an optional implementation, the preset maximum list length can be set to 128 or 512 depending on the data skewness.

[0081] The coordinating node aggregates all cross-node near-duplicate candidate pairs with the local same-bucket near-duplicate candidate pairs reported by each storage node, removes duplicate pairs, and forms a global candidate pair set. The method for removing duplicate pairs is as follows: the two record identifiers in each candidate pair are arranged in lexicographical order and concatenated into a composite key. All composite keys are then inserted into a hash set to automatically remove duplicates.

[0082] The refinement and verification phase is also scheduled by the coordinating node. The coordinating node sends back the record identifiers for each candidate pair in the global candidate pair set to its respective storage node, requesting the nearest storage processing unit of each storage node to construct an Odd Sketch bit vector for the corresponding original data record. Each storage node retrieves the original data record from the storage medium based on the record identifier, constructs an Odd Sketch bit vector for the attribute element set of each record, and then uploads the bit vector along with the record identifier to the coordinating node. Since the Odd Sketch bit vector is fixed at 256 bits (32 bytes) in this embodiment, and each candidate pair requires uploading two Odd Sketch bit vectors totaling 64 bytes, even if the global candidate pair set contains 1 million candidate pairs, the total transmission volume is only about 61 megabytes, within the bandwidth capacity range of the inter-storage node interconnection channel.

[0083] The coordinating node performs a bitwise XOR operation on the Odd Sketch bit vectors of two records in the same candidate pair to obtain a difference bit vector. The number of bits with a value of 1 in the difference bit vector is then counted to obtain a parity difference count. This operation is implemented on the coordinating node using bit manipulation instructions. For a 256-bit bit vector, the XOR operation can be completed in a single clock cycle using a 256-bit wide SIMD instruction. The number of bits with a value of 1 is then counted using a popcount instruction. Candidate pairs with a parity difference count greater than or equal to a preset refinement threshold are removed. After Odd Sketch refinement, the number of remaining candidate pairs is typically reduced to 30% to 50% of the original global candidate pair set.

[0084] The remaining candidate pairs enter the field-level precise comparison stage. The coordinating node requests all attribute field contents of the two original data records in the candidate pair from the corresponding storage node. This step involves the transmission of actual original data, but since the number of candidate pairs has been significantly reduced after the first two rounds of screening, the transmission volume is controllable. The near-storage processing unit of each storage node reads the corresponding record from the storage medium and uploads it to the coordinating node. The coordinating node performs a byte-level equality comparison of the attribute fields of the two records column by column according to the same column order, and the proportion of attribute fields with completely identical content to the total number of attribute fields is used as the field consistency rate. ,in The value range is from 0 to 1. The field consistency rate... Greater than or equal to the preset precise confirmation threshold Candidate pairs were confirmed as verified near-duplicate record pairs, among which To preset the accuracy confirmation threshold, this embodiment uses 0.8. In an optional implementation, field-level precise comparison can also be replaced by edit distance comparison or weighted Jaccard similarity calculation, which is suitable for scenarios where there are subtle character differences in the content of attribute fields.

[0085] The global resolution phase is the final step in the entire data cleaning process and the step that actually produces the data deletion effect. The coordinating node constructs a near-duplicate graph using the record identifiers of all verified near-duplicate record pairs as nodes and the pairing relationships as edges. The near-duplicate graph is an undirected graph where each node represents one original data record and each edge represents a set of verified near-duplicate relationships. The coordinating node performs a connected component search on the near-duplicate graph, dividing all nodes into several disjoint connected components, each forming a near-duplicate cluster. The connected component search can be implemented using a disjoint-set data structure: initially, each node forms its own independent set; all edges are traversed, and a merge operation is performed on the two endpoints of each edge. The merge operation in the disjoint-set data structure uses a rank-based merging and path compression strategy, making... Nodes The time complexity of searching for the connected components of a graph with edges is O(n). ,in This represents the total number of nodes in the graph. The total number of edges in the graph. For the inverse Ackermann function, for any practically possible value, Since the value is no more than 4, it can be considered approximately linear. In an optional implementation, when the number of verified near-duplicate record pairs exceeds 10 million, a block-parallel strategy can be adopted. The near-duplicate graph is divided into multiple subgraphs according to the record identifier range, and connected component search is performed separately for each subgraph. Then, the connected component results of each subgraph are merged again at the global level.

[0086] For each near-duplicate cluster, the coordinating node needs to determine which records to retain and which to discard. The coordinating node sends all record identifiers within each near-duplicate cluster to the corresponding storage node, requesting the data write timestamp for each record. Within the same near-duplicate cluster, the data write timestamps of all records are compared, and the record with the latest data write timestamp is selected as the retained record. If two or more records within the same near-duplicate cluster have identical data write timestamps, their record identifiers are further compared, and the record with the largest lexicographical order of its identifier is selected as the retained record to ensure the determinism and reproducibility of the resolution result. The remaining records are marked as pending discard.

[0087] The coordinating node distributes the retained record identifiers and expiring record identifiers for all near-duplicate clusters to the corresponding storage nodes. This distribution process is grouped by storage node, meaning each storage node only receives resolution instructions related to data records on its own node, thereby reducing unnecessary communication. The near-storage processing unit of each storage node registers the storage address and record identifier of the retained records in a local cleansing result index table. The local cleansing result index table is organized using a B+ tree structure, with the record identifier as the key and the storage block address as the value, supporting fast retrieval of valid records by the subsequent data lake query engine. In an optional implementation, the local cleansing result index table can also use a hash index structure, suitable for access modes primarily based on exact match queries.

[0088] The storage block addresses occupied by all records awaiting erasure are written to the local space reclamation queue. The local space reclamation queue follows a first-in, first-out (FIFO) structure. During idle periods, the storage media controller retrieves the storage block addresses from the head of the queue and performs erasure and release. An idle period refers to the time during which the storage media controller is not currently processing any host-side read / write requests. Scheduling the erasure operation during the idle period avoids competition between the erasure process and normal business I / O. In high-load scenarios, if the idle period is insufficient to handle all backlogged reclamation requests, the storage media controller can postpone the reclamation operation to the next idle period without affecting the correctness of the erasure results.

[0089] After completing the cleanup of all near-duplicate clusters on its local node, each storage node sends a local cleanup completion receipt to the coordinating node. The receipt includes a statistical summary of the cleanup, including the total number of records retained, the total number of records evicted, and the number of bytes of storage space released on the local node. Once the coordinating node receives all local cleanup completion receipts from all storage nodes, it generates a global cleanup completion identifier. This global cleanup completion identifier is atomically written to the data lake's metadata service, ensuring that any query engine sees the data before cleanup and only sees the cleaned data after reading this identifier, thus guaranteeing consistency between the cleanup process and external queries.

[0090] The coordinating node synchronously updates the mapping relationship between the number of valid records and storage addresses for the corresponding data partition in the data lake's global metadata index based on the local cleansing result index table reported by each storage node. The data lake's global metadata index is typically implemented using a distributed key-value store, such as Apache HBase or a self-developed metadata service. The coordinating node submits a batch of update requests to the metadata service. Each update request includes the data partition identifier, the updated number of valid records, and the storage address range of valid records on each storage node. The metadata service processes these update requests in a transactional manner, ensuring that the metadata for all partitions is either fully updated successfully or fully rolled back.

[0091] Regarding the system-level fault tolerance and recovery mechanism, this embodiment is designed as follows: During the local processing phase or the cross-node aggregation phase, if a storage node fails, the coordinating node marks the node as unavailable after a preset timeout period and continues cleaning based on the data from the remaining storage nodes. The preset timeout period is set to 60 seconds in this embodiment. After the failed node recovers, it can re-participate in the next cleaning cycle. During the global resolution phase, if the coordinating node itself fails, since the original data records on each storage node have not been actually deleted (the erase operation is still queued in the space reclamation queue), after the failure recovers, the coordinating node can re-execute global resolution from the output results of the refinement and verification phase, without causing data loss. In an optional implementation, a backup coordinating node can be set up, which takes over the workflow when the primary coordinating node fails.

[0092] Regarding system throughput, three key factors affect the cleaning performance: the read bandwidth of storage nodes, the network bandwidth of the interconnection channels between storage nodes, and the computing power of the coordinating node. During the local processing phase, each storage node works independently in parallel, and the system read bandwidth scales linearly with the number of storage nodes. The data transfer volume during the cross-node aggregation phase is minimal and does not constitute a performance bottleneck. The data transfer volume during the refinement and verification phase is proportional to the size of the global candidate pair set; in extreme cases (such as when the data duplication rate exceeds 50%), it may generate high network traffic. The computational cost of the coordinating node performing Odd Sketch bitwise XOR, field-level alignment, and connected component search is proportional to the size of the global candidate pair set and the size of the verified near-duplicate record pairs. In a test scenario involving 8 storage nodes, a total of 800 million records, and a data duplication rate of approximately 5%, the entire cleaning process took approximately 14 minutes, with the local processing phase accounting for approximately 62%, cross-node aggregation and intersection operations accounting for approximately 8%, the refinement and verification phase accounting for approximately 22%, and global resolution and metadata updates accounting for approximately 8%.

[0093] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A near-memory processing unit for data cleaning, characterized in that: Integrated into the storage media controller layer, including: The signature generation module includes a multi-way hash calculation channel and an on-chip accumulation register group, which is used to generate SimHash fixed-length binary signatures for the original data records to be cleaned; The segment partitioning and bucket mapping module includes a bucket mapping table storage area and a local candidate pair buffer. It is used to divide the SimHash fixed-length binary signature into multiple segments, establish a bucket mapping table with the segment identifier as the key, filter local near-duplicate candidate pairs in the same bucket according to the bucket mapping table, and extract the local segment identifier set for each segment position. The Theta Sketch encoding module is used to construct the Theta Sketch structure from the local tape identifier set at each tape segment position and establish a reverse mapping relationship between the normalized hash value and the original tape identifier. The Odd Sketch build module is used to construct Odd Sketch bit vectors from the set of attribute elements of the original data record; The cleaning and disposal module includes a local cleaning result index table and a local space reclamation queue, which are used to register the storage address of retained records and reclaim the storage space of records to be eliminated.

2. The near-memory processing unit according to claim 1, characterized in that, The original data record is a line-based text data record. The process of generating a SimHash fixed-length binary signature by the signature generation module includes: splitting each original data record into multiple attribute fields according to column delimiters; performing sliding window segmentation on each attribute field according to a fixed byte length to obtain a sequence of feature fragments; sending each feature fragment into a multi-way hash calculation channel; each hash calculation channel using its own independent preset hash seed to perform hash operation on the feature fragment and outputting a binary intermediate vector of the same length as the target signature; performing bit-by-bit accumulation on all binary intermediate vectors generated by all feature fragments in the same original data record in the on-chip accumulation register group; after accumulation, performing sign determination on the accumulated value at each bit position; setting the position with a positive accumulated value to 1, and setting the position with a zero accumulated value to 0, thus obtaining the SimHash fixed-length binary signature.

3. The near-memory processing unit according to claim 1, characterized in that, The working process of the segment partitioning and bucket mapping module includes: dividing the SimHash fixed-length binary signature into multiple continuous and non-overlapping segments along the bit direction. Each segment consists of a fixed number of continuous bits. The bit sequence of each segment is used as the segment identifier. A bucket mapping table is established on-chip for each segment position. The bucket mapping table uses the segment identifier as the key and the record identifier list as the value. The segment identifier of each original data record at each segment position is used as the index key. The corresponding record identifier is appended to the record identifier list of the corresponding segment identifier in the bucket mapping table. During the appending process, the number of existing record identifiers in the record identifier list is checked. When the number of existing record identifiers is greater than or equal to 1, the current record identifier is paired one by one with all existing record identifiers in the record identifier list to form local near-duplicate candidate pairs in the same bucket and written to the local candidate pair buffer.

4. The near-memory processing unit according to claim 1, characterized in that, The process of constructing the ThetaSketch structure by the Theta Sketch encoding module includes: applying a uniform hash transformation to each tag in the local tag set to map it to an open interval of 0 to 1 to obtain a normalized hash value; inserting all normalized hash values ​​one by one into an on-chip ordered array with a preset fixed capacity and maintaining ascending order; when the ordered array is full and a newly arrived normalized hash value is less than the last element of the array, replacing the last element with the new value and reordering; and discarding a newly arrived normalized hash value that is greater than or equal to the last element of the array. After all tags have been processed, the ordered array is the local Theta Sketch structure. The largest normalized hash value in the ordered array is recorded as the local truncation threshold, and a reverse mapping relationship is established between each normalized hash value and the original tag.

5. The near-memory processing unit according to claim 1, characterized in that, The process of constructing an OddSketch bit vector using the Odd Sketch building module includes: taking the complete byte content of each attribute field of the original data record as an attribute element; forming an attribute element set by combining all attribute elements of the same original data record; initializing a fixed-length all-zero bit vector on-chip; calculating a bit index value for each attribute element in the attribute element set using a preset hash function; locating the bit pointed to by the bit index value in the bit vector and performing a flip operation, i.e., setting the bit to 1 if the current value of the bit is 0 and setting it to 0 if the current value of the bit is 1; and finally, the bit vector obtained after processing all attribute elements is the Odd Sketch bit vector.

6. A data lake cleaning system, characterized in that, include: Multiple storage nodes, each storage node has a near storage processing unit for data cleaning. The near storage processing unit is integrated into the storage medium controller layer and is used to generate SimHash fixed-length binary signatures for the original data records and perform band segmentation and in-band candidate pair screening, construct Theta Sketch structure and construct Odd Sketch bit vectors. The coordinating node communicates with all storage nodes and is used to receive the ThetaSketch structure from each storage node according to the strip position and perform cross-node set intersection operations to form a global candidate pair set. Based on the Odd Sketch bit vector, it performs refined filtering and field-level precise comparison to confirm verified near-duplicate record pairs. Based on the verified near-duplicate record pairs, it constructs a near-duplicate graph and performs connectivity component-level resolution and data lake global metadata index update.

7. The data lake cleaning system according to claim 6, characterized in that, The process of cross-node set intersection operation includes: the coordinating node takes the minimum value of the local truncation thresholds collected from all storage nodes for the same strip position as the global truncation threshold, retains only the normalized hash value entries in each local Theta Sketch structure that are strictly less than the global truncation threshold and merges them into a global Theta Sketch structure, filters out the normalized hash value entries that are simultaneously contained in two or more different storage nodes as cross-node co-occurrence entries, restores the cross-node co-occurrence entries to the corresponding strip identifiers through the reverse mapping relationship, locates the corresponding record identifier list through the strip bucket mapping table, pairs the record identifiers from different storage nodes to form cross-node near-duplicate candidate pairs, summarizes all cross-node near-duplicate candidate pairs and local same-bucket near-duplicate candidate pairs and removes duplicate pairs to form a global candidate pair set.

8. The data lake cleaning system according to claim 6, characterized in that, The process of refined filtering and field-level precise comparison includes: the coordinating node performs a bitwise XOR operation on the Odd Sketch bit vectors of two records in the same candidate pair to obtain the difference bit vector; the number of bits with a value of 1 in the difference bit vector is counted to obtain the parity difference count; candidate pairs with parity difference counts greater than or equal to the preset refined filtering threshold are removed; for candidate pairs with parity difference counts less than the preset refined filtering threshold, all attribute fields of the two records are compared column by column in the same column order to obtain the byte-level equality comparison; the proportion of attribute fields with completely identical content to the total number of attribute fields is counted as the field consistency rate; and candidate pairs with a field consistency rate greater than or equal to the preset precise confirmation threshold are confirmed as verified near-duplicate record pairs.

9. The data lake cleaning system according to claim 6, characterized in that, The process of connectivity component-level resolution includes: The coordinating node constructs a near-duplicate graph using the record identifiers of all verified near-duplicate record pairs as nodes and the pairing relationships as edges. It performs a connectivity component search on the near-duplicate graph, dividing all nodes into several disjoint connected components. Each connected component forms a near-duplicate cluster. The coordinating node sends all record identifiers within each near-duplicate cluster to the corresponding storage node, requesting the return of the data write timestamp for each record. Within the same near-duplicate cluster, it compares the data write timestamps of all records, selecting the record with the latest data write timestamp as the retained record, and marking the remaining records as records to be evicted. Each storage node... The near-storage processing unit of each storage node registers the storage address and record identifier of the retained records in the local cleaning result index table. It writes the storage block addresses occupied by all records to be eliminated into the local space reclamation queue, which is then erased and released by the storage media controller during idle periods. After the elimination operation is completed, each storage node sends a local cleaning completion receipt to the coordinating node. After receiving the local cleaning completion receipts from all storage nodes, the coordinating node generates a global cleaning completion identifier and synchronously updates the mapping relationship between the number of valid records and storage addresses of the corresponding data partition in the global metadata index of the data lake based on the local cleaning result index table reported by each storage node.

10. The data lake cleaning system according to claim 6, characterized in that, Each storage node is connected to the coordinating node through the inter-storage node interconnection channel. The original data records are line-based text data records. The near storage processing unit reads the original data records one by one from the read buffer of the storage medium controller and performs processing.

Images