Method and system for quickly removing duplicate of document based on multi-algorithm fusion

Abstract

The invention discloses a rapid document duplicate removal method and system based on multi-algorithm fusion, and the method comprises the steps: carrying out the preprocessing of a document, and generating a duplicate removal vocabulary set; performing rapid member judgment on the de-duplicated vocabulary set through a Bloom filter; generating a locality sensitive hash fingerprint and a probability signature in parallel for the document passing through the Bloom filter; constructing a zoning locality sensitive hash index, and retrieving a candidate similar document set from a document library; calculating the similarity of the title dimension, the content dimension and the metadata dimension for each candidate similar document, and performing weighted fusion on the similarity of each dimension according to a preset weight to obtain a comprehensive similarity score corresponding to the document; and comparing the comprehensive similarity score with a set similarity threshold value to judge whether the document is repeated or not according to a comparison result. Through a collaborative mechanism of coarse screening, indexing and similarity fusion decision, the calculation complexity is remarkably reduced, and meanwhile, the accuracy of large-scale document deduplication is improved.

Description

Technical Field

[0001] This invention relates to the field of document deduplication technology, and in particular to a fast document deduplication method and system based on multi-algorithm fusion. Background Technology

[0002] With the deepening of digital office and information technology construction, the number of electronic documents generated by various organizations is growing exponentially. According to statistics, enterprises generate millions of documents on average each year, of which 15%-30% are duplicates or highly similar. These redundant documents not only waste storage resources, but also seriously affect information retrieval efficiency, knowledge management quality, and decision-making accuracy. Therefore, efficient and accurate document deduplication technology has become a key requirement for modern information management systems.

[0003] Currently, mainstream document deduplication methods are mainly divided into two categories: methods based on exact matching and methods based on similarity calculation. The former (such as MD5 and SHA1 hashing) can only identify documents with completely identical content, and is completely ineffective for common scenarios such as format differences, paragraph adjustments, or semantic rewriting. The latter, while able to identify semantically similar documents, generally suffers from high computational complexity, large memory consumption, and slow processing speed, making it difficult to meet the practical needs of large-scale applications. Specifically, existing similarity-based deduplication technologies face the following key technical bottlenecks: First, each algorithm has inherent limitations. For example, the SimHash algorithm performs poorly on short texts or documents with sparse word distribution; the edit distance algorithm has a time complexity of O(mn), making it unsuitable for massive amounts of documents; the TF-IDF method requires storing the complete document-term matrix, resulting in huge memory overhead; while MinHash is suitable for set similarity estimation, its computational and storage costs are high, making real-time processing difficult. These algorithms are each suitable for specific scenarios, but they cannot comprehensively cover diverse document types and deduplication requirements.

[0004] Secondly, traditional deduplication processes generally employ a pairwise comparison strategy, calculating the similarity of every pair of documents in the document library, resulting in an overall time complexity as high as O(n²). With millions or even tens of millions of documents, this strategy can take several days to compute, severely limiting the system's practicality and scalability. More importantly, existing methods lack an efficient candidate document selection mechanism, wasting significant computational resources on clearly dissimilar document pairs, leading to serious performance waste.

[0005] In summary, there is an urgent need for a document deduplication method that can overcome the bottleneck of O(n²) computational complexity and integrate the advantages of multiple algorithms to improve accuracy. Summary of the Invention

[0006] In view of this, this invention proposes a fast document deduplication method and system based on multi-algorithm fusion, which can improve the accuracy and efficiency of document deduplication. This invention provides the following technical solution: A fast document deduplication method based on multi-algorithm fusion includes the following steps: Standardized preprocessing is performed on the input document to extract the title, body content and metadata, and a deduplicated vocabulary set is generated based on the body content to generate structured text features; The deduplication vocabulary set is quickly determined by a Bloom filter to exclude absolutely non-duplicate documents. For documents that pass through the Bloom filter, generate locality-sensitive hash fingerprints based on semantic distribution; A segmented local sensitive hash index is constructed based on the local sensitive hash fingerprint, and a set of candidate similar documents is retrieved from the document library using the segmented local sensitive hash index; For the set of candidate similar documents, the similarity of the title dimension, content dimension and metadata dimension is calculated for each candidate similar document, and the similarity of each dimension is weighted and fused according to the preset weight to obtain the comprehensive similarity score of the document. The overall similarity score is compared with a set similarity threshold to determine whether the documents are duplicates based on the comparison results.

[0007] Optionally, the standardization preprocessing includes: The input document is parsed to extract the title text, body content, and metadata; The extracted title and body texts are cleaned to remove HTML tags, special characters, and non-standard whitespace characters. The cleaned title and body texts are segmented and stop words are filtered to generate a title vocabulary list, a body vocabulary list, and a deduplicated vocabulary set. Based on the title vocabulary list and the body vocabulary list, TF-IDF vectors are constructed and normalized to serve as structured text features.

[0008] Optionally, the method further includes: generating a probabilistic signature based on the set of deduplicated words to which the document belongs while generating the locality-sensitive hash fingerprint; The Local Sensitive Hash fingerprint is a SimHash fingerprint, which generates a 64-bit binary fingerprint based on the TF-IDF vector. The probabilistic signature is a MinHash signature, which is based on the deduplicated vocabulary set and generates a signature vector composed of multiple MinHash values ​​through multiple independent hash functions.

[0009] Optionally, constructing a segmented locality-sensitive hash index based on the locality-sensitive hash fingerprint includes: The local sensitive hash fingerprint is divided into a preset number of bands, and each band contains a corresponding number of consecutive binary bits; For each band, calculate its band hash value and map the band hash value to the identifier of the corresponding document in the global inverted index; Based on the mapping relationship between the zoned hash values ​​of all documents and the document identifiers, the zoned local sensitive hash index is constructed.

[0010] Optionally, the similarity in the title dimension is obtained by calculating and normalizing the Hamming distance between the SimHash title fingerprints of the target document and the candidate document; The content dimension similarity is obtained by comparing the number of times the hash values ​​at the same position in the MinHash signatures of the target document and the candidate document are equal, and then dividing the number by the total length of the MinHash signature; The similarity of the metadata dimension is obtained by calculating the similarity of each metadata item of the target document and the candidate document separately, and then taking a weighted average of the similarity of each metadata item or combining them according to a preset rule.

[0011] Optionally, when retrieving the candidate similar document set through the segmented locality-sensitive hash index, each segmented hash value of the target document is used as a query key to find the corresponding document identifier list in the segmented locality-sensitive hash index, and a union operation is performed on all the retrieved document identifier lists to obtain the candidate similar document set.

[0012] This invention further discloses a fast document deduplication system based on multi-algorithm fusion, comprising: The preprocessing module is used to perform standardized preprocessing on the input document, extract the title, body content and metadata, and generate a deduplicated vocabulary set based on the body content to generate structured text features; The Bloom filter module is used to quickly determine the membership of the deduplication vocabulary set and exclude absolutely non-duplicate documents; The multi-feature generation module is used to generate, in parallel, local sensitive hash fingerprints based on semantic distribution and probabilistic signatures based on deduplicated vocabulary sets from documents that have passed through the Bloom filter. The index building and retrieval module is used to build a segmented local sensitive hash index based on the local sensitive hash fingerprint, and retrieve a set of candidate similar documents from the document library through the segmented local sensitive hash index; The similarity calculation module is used to calculate the similarity of the title dimension, content dimension and metadata dimension for each candidate similar document in the candidate similar document set, and to perform weighted fusion of the similarity of each dimension according to the preset weight to obtain the comprehensive similarity score of the document. The duplicate determination module is used to compare the comprehensive similarity score with a set similarity threshold to determine whether the document is duplicated based on the comparison result.

[0013] The present invention further discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0014] The present invention further discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method.

[0015] The present invention further discloses a computer program product, including a computer program that implements the above-described method when executed by a processor.

[0016] According to the technical solution of the present invention, a complete technical solution is adopted, which includes standardized preprocessing, coarse screening with Bloom filters, parallel generation of local sensitive hash fingerprints and probabilistic signatures, construction of a segmented local sensitive hash index to retrieve candidate sets, multi-dimensional similarity weighted fusion, and threshold determination. This solution effectively solves the problems of low computational efficiency, insufficient accuracy, and high resource consumption faced by traditional document deduplication methods in large-scale scenarios. Among them, the Bloom filter quickly eliminates absolutely non-duplicate documents with extremely low overhead, significantly reducing the amount of subsequent computation. The segmented LSH index based on SimHash fingerprints reduces the complexity of similar document retrieval from O(n²) to approximately O(1), achieving sub-second response for tens of millions of documents. Furthermore, the similarity of the candidate set is calculated and weighted for fusion in the three dimensions of title, content, and metadata, overcoming the semantic blind spot of a single algorithm and significantly improving the recognition accuracy of complex repetitive situations such as semantic rewriting and format differences, while taking into account high efficiency, high accuracy, and low resource consumption. Attached Figure Description

[0017] For illustrative and not limiting purposes, the present invention will now be described in conjunction with embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the document deduplication method based on multi-algorithm fusion in an embodiment of the present invention. Figure 2 This is a schematic diagram of the document deduplication method architecture based on multi-algorithm fusion in an embodiment of the present invention; Figure 3This is a schematic diagram of the structure of the document fast deduplication system based on multi-algorithm fusion in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the electronic device in an embodiment of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0019] It should be noted that, where there is no conflict, the embodiments and features of the embodiments in this application can be combined with each other. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0020] refer to Figure 1 and Figure 2 This implementation discloses a fast document deduplication method based on multi-algorithm fusion. Its core lies in constructing a structured, hierarchical, and collaborative processing architecture. Through clear data flow connections and functional module collaboration, it overcomes the traditional O(n²) computation bottleneck while ensuring high accuracy, achieving efficient deduplication of large-scale documents. The architecture is attached. Figure 2 As shown, it is divided into three tightly coupled levels: "file preprocessing," "fusion calculation," and "similarity calculation." Specifically, it includes the following steps: S100: Perform standardized preprocessing on the input document to extract the title, body content, and metadata to generate structured text features.

[0021] First, the system receives raw documents from various sources, in formats including but not limited to TXT, PDF, DOC, DOCX, or HTML. The preprocessing module first parses the documents, using appropriate parsers such as Apache PDFBox for PDF, POI for DOC / DOCX, and Jsoup for HTML, to accurately extract the document's title text, body content, and key metadata, including author, creation time, file type, and tags. This removes irrelevant formatting and style information, ensuring that subsequent processing focuses on the content itself.

[0022] Subsequently, a deep cleaning operation was performed on the extracted title text and body content: removing residual HTML / XML tags (such as...). , etc.), special symbols, control characters, zero-width characters, and non-standardized whitespace characters (such as multiple consecutive spaces, tabs, etc.) to obtain a pure text containing only readable text and necessary punctuation.

[0023] Next, perform language unit segmentation, i.e., word segmentation, on the cleaned title text and the pure text of the body content. For Chinese text, use a word segmentation tool based on a dictionary or deep learning (such as jieba, HanLP) to segment the continuous text into a sequence of meaningful words; for English text, segment it naturally by spaces and punctuation. After word segmentation, further perform stop word filtering to remove high-frequency but low-semantic contribution words, such as "的", "了", "在" in Chinese, "the", "is", "and" in English, etc., as well as invalid character combinations that are too short, so as to retain the keyword vocabulary carrying core semantic information, and generate a title vocabulary list, a body vocabulary list, and the corresponding deduplicated vocabulary set.

[0024] Based on this, the system constructs TF-IDF vectors respectively based on the title vocabulary list and the body vocabulary list: First, count the term frequency (TF) of each word in the document, and calculate its weight in combination with the word length. The calculation formula is: weight = TF×log(1 + len(word)). Subsequently, organize the word weights into a high-dimensional sparse vector and perform L2 normalization so that the vector norm is 1, which is convenient for subsequent similarity calculation. Finally, the preprocessing module outputs a structured text feature object, in which the standardized data of the title dimension, the body content dimension, and the metadata dimension are clearly separated and encapsulated, including the title TF(-IDF) vector (title.tfidf_vector) and the body TF(-IDF) vector (content.tfidf_vector), providing a unified and high-quality input basis for subsequent algorithms such as Bloom filters, SimHash, and MinHash.

[0025] S200: Quickly determine the membership of the vocabulary set of the document through a Bloom filter and exclude definitely non-duplicate documents.

[0026] Specifically, a global Bloom filter instance is first initialized, consisting of a fixed-length bit array and a set of independent hash functions. When a new document undergoes normalization preprocessing, its output "deduplicated vocabulary set" (i.e., `content.word_set`, containing all unique words in the document after stop word removal) is fed into the Bloom filter module for evaluation. The module iterates through each word in the vocabulary set, using all the hash functions to calculate its mapping position in the bit array. If all words have a 1 in the corresponding bit of all hash functions, the Bloom filter determines that the document "may already exist," allowing it to proceed to the subsequent feature generation and index building process; conversely, if any word has a 0 in the corresponding bit of any hash function, it can be determined that the document has absolutely never appeared in the system and is a non-duplicate document.

[0027] For documents determined to be non-duplicate, their subsequent processing is immediately terminated, and high-overhead operations such as SimHash fingerprint generation, MinHash signature calculation, and LSH index construction are no longer performed, thus significantly saving computing resources. Experiments show that in typical enterprise document scenarios, this Bloom filter coarse screening layer can effectively eliminate more than 90% of obviously non-duplicate documents, reducing the processing load of subsequent algorithms by more than an order of magnitude.

[0028] It is worth noting that while Bloom filters can produce some false positives (i.e., incorrectly classifying documents as "possibly existing"), they do not produce false negatives (i.e., they will not miss documents as "absolutely non-existent"). All documents judged as "possibly existing" still need to be finally confirmed through subsequent LSH indexing and multi-dimensional similarity calculations, thereby achieving a balance between efficiency and accuracy in the overall process while ensuring high recall.

[0029] S300: For documents that have passed through the Bloom filter, generate in parallel a locally sensitive hash fingerprint based on semantic distribution and a probabilistic signature based on a vocabulary set.

[0030] For documents initially identified as "potentially existing" by the Bloom filter, two complementary feature generation operations are performed in parallel: first, generating a locally sensitive hash fingerprint based on semantic distribution, i.e., SimHash fingerprint; and second, generating a probabilistic signature based on a vocabulary set, i.e., MinHash signature. This process fully utilizes the structured text features output by the preprocessing module to achieve efficient and loosely coupled parallel computation.

[0031] The SimHash fingerprint generation process is as follows: First, obtain the standardized data output after preprocessing, the TF-IDF vector of the main text (content.tfidf_vector). Then, for each word in the vector... Calculate its weighted weights: ,in, For vocabulary TF-IDF vector, For vocabulary The length of each word. Use the MD5 hash function to map it to a 64-bit binary vector. Next, the vectors of all words are weighted and superimposed according to their weights. Finally, each sign bit (0 or 1) of the superimposed result is used as the 64-bit SimHash fingerprint of the document. This fingerprint is local sensitive, that is, documents with similar semantics have a small Hamming distance in their SimHash fingerprints.

[0032] The process of generating a MinHash signature is as follows: Obtain the preprocessed deduplicated vocabulary set (content.word_set), which is the unique set of all words in the document after stop word removal. Then, initialize 128 independent hash functions. For each hash function, iterate through all words in the vocabulary set, calculate its hash value, and record the minimum value. Arrange these 128 minimum hash values ​​sequentially to form a 128-dimensional integer vector, which is the MinHash signature (minhash_signature) of the document. This signature can preserve the Jaccard similarity relationship between the original vocabulary sets with high probability and is insensitive to word order, making it suitable for large-scale set similarity estimation.

[0033] The generation of SimHash fingerprints and MinHash signatures are executed in complete parallel: SimHash computation relies on TF-IDF vectors, focusing on capturing the overall semantic distribution of the document; MinHash computation relies on a vocabulary set, focusing on capturing the degree of word overlap. Their input data sources are different, and their computational logic is independent, allowing them to be performed simultaneously in multi-threaded or heterogeneous computing units without blocking each other. The generated SimHash fingerprint is used for subsequent LSH index construction and fast retrieval, while the MinHash signature is stored in the storage layer for content-dimensional similarity calculations during the actuarial stage. This parallel fusion mechanism not only improves processing throughput but also provides a complementary feature foundation for subsequent multi-dimensional decision-making.

[0034] S400: Construct a segmented local sensitive hash index based on the local sensitive hash fingerprint, and retrieve a set of candidate similar documents from the document library through the segmented local sensitive hash index.

[0035] First, the local sensitive hash fingerprint generated by the SimHash algorithm after filtering by the Bloom filter in step S300 is obtained, namely the SimHash fingerprint. In order to support efficient approximate nearest neighbor retrieval, the system performs banding processing on the fingerprint: the 64-bit binary fingerprint is evenly divided into 16 consecutive bands, each band containing 4 binary bits (i.e., the bit interval [4i, 4i+4] corresponding to the i-th band, where i=0,1,...,15).

[0036] Subsequently, a band hash value (band_hash_i) is calculated for each band. Specifically, each 4-bit binary string is directly interpreted as an integer between 0 and 15 (or further mapped to a 32-bit integer using a lightweight hash function such as MurmurHash3), serving as a unique identifier for that band.

[0037] A global inverted index structure (global_lsh_index) is maintained, where the key is a list of all possible band hash values ​​and the value is a list of document identifiers (doc_id). For the currently processed document, the system uses its 16 band hash values ​​as keys and appends the document's doc_id to the document list corresponding to the key. This means that a single document can be recorded under a maximum of 16 different index entries. This index structure is continuously built and persistently stored during the document ingestion phase. As the document database grows, the index automatically expands, but each insertion operation only involves 16 hash table updates, with a time complexity of O(1), demonstrating good scalability.

[0038] When deduplication of a target document is required, its SimHash fingerprint is first calculated, and the same banding process as in the index building phase is performed to obtain its 16 banded hash values. Next, these 16 banded hash values ​​are used as query keys to search the constructed global banded locally sensitive hash index. For each query key, if a corresponding list of document IDs exists in the index, it is added to the temporary result set.

[0039] Finally, a union operation is performed on all retrieved document ID lists, merging all lists and removing duplicate document IDs to obtain the final set of candidate similar documents. This set contains only those documents that completely match the target document in at least one band.

[0040] This step achieves the following: if the SimHash fingerprints of two documents have a small Hamming distance, the probability of them matching simultaneously across multiple bands is significantly higher than that of random documents. Therefore, by adopting the strategy of "including any matching band as a candidate", the size of the candidate set can be reduced from the total number of documents n in the database to a very small constant while ensuring high recall.

[0041] S500: For the candidate similar document set, calculate the similarity of the title dimension, content dimension and metadata dimension respectively, and perform weighted fusion of the similarity of each dimension according to the preset weight to obtain the comprehensive similarity score.

[0042] First, load the preprocessed results (ProcessedDocument) of the target document (target_doc) and each candidate document (candidate_doc) and the pre-computed algorithm features (AlgorithmResults) from storage, including: SimHash title fingerprint (title_fingerprint), MinHash content signature (content_signature), and normalized metadata (metadata).

[0043] Regarding the calculation of title similarity (title_sim): Extract the SimHash title fingerprints (both 64-bit binary strings) from the target document and candidate documents, and calculate the Hamming distance (hamming_dist) between them, which is the number of different bits. Then, normalize this distance to a similarity value in the [0,1] interval. ; For content-dimensional similarity calculation (content_sim): Extract the MinHash signatures (both 128-dimensional integer vectors) of the target document and candidate documents, and compare the number of identical hash values ​​at the same positions in the two signatures (match_count). Content similarity is defined as: ; Similarity calculation for metadata dimension (meta_sim): For each metadata item of the target document and candidate documents, taking author, tags, and creation time as examples, calculate the similarity: For collection-type fields such as authors or tags, Jaccard similarity is used for calculation; For creation time, calculate the time difference (e.g., number of days) and apply it using a normalization function. Convert to similarity; For other text-based metadata (such as source and department), edit distance and normalization can be used.

[0044] Subsequently, the similarity of each metadata item is weighted and averaged or combined according to preset rules or logical rules. For example, if the authors are the same and the time difference is less than 7 days, then meta_sim=1; otherwise, the average value is taken to obtain the final meta_sim.

[0045] Finally, a multi-dimensional weighted fusion and comprehensive score calculation are performed: The similarity scores across the three dimensions are linearly weighted and fused according to preset weights to calculate the overall similarity score (Overall_Score): Among them, each weight satisfies .

[0046] This implementation provides an exemplary weight allocation: The value is set to 0.5 to ensure that the content is the core basis for judgment; The value is set to 0.3 to help ensure the title maintains thematic consistency; The value is set to 0.2 to use metadata as a supplementary reference.

[0047] S600: Compare the overall similarity score with the set similarity threshold to determine whether the documents are duplicates based on the comparison result.

[0048] Specifically, a three-tiered threshold determination mechanism is pre-configured, including two key thresholds: a high threshold (high_threshold) and a low threshold (low_threshold). By default, high_threshold is set to 0.8 and low_threshold to 0.5. This threshold system is not dynamically generated but can be manually set according to business scenarios or pre-configured by document type, ensuring the controllability and interpretability of the determination logic.

[0049] This implementation provides an exemplary logic rule for determining duplicates: 1. If the overall similarity score is ≥ high_threshold: The target document is determined to be a duplicate of the candidate document. This determination has a high confidence level, and deduplication operations (such as marking, archiving, or deleting) can be performed directly without manual intervention.

[0050] 2. If low_threshold ≤ overall similarity score <high_threshold: The target document is deemed potentially duplicated. While these documents may not meet explicit duplication criteria, they exhibit significant similarity and may contain semantic rewriting, partial citations, or templated content. These results are marked as "Pending Review" and pushed to the manual review queue for further assessment by business personnel, balancing automation efficiency with business accuracy.

[0051] 3. If the overall similarity score is considered <low_threshold: If the document is determined to be a non-duplicate document, the processing flow for that candidate document ends.

[0052] The three-level judgment mechanism in this embodiment effectively balances the contradiction between recall and precision: the high threshold ensures the reliability of automatic deduplication results and avoids accidental deletion; the middle interval introduces a manual review step to prevent missed detections due to algorithm limitations; and the low threshold quickly excludes obviously irrelevant documents, improving overall processing efficiency.

[0053] Furthermore, the system supports adjusting the threshold configuration based on document type or business scenario. For example, in academic paper plagiarism detection, to reduce the risk of missed detections, `high_threshold` can be appropriately lowered to 0.75; while in deduplication of internal corporate documents, to avoid false positives, `high_threshold` can be increased to 0.85. However, it should be emphasized that the threshold itself is a fixed value in a single judgment process, and its adjustment is an operation at the system configuration level, not a dynamic calculation result during algorithm runtime.

[0054] Finally, the judgment results, along with the similarity scores of each dimension, the comprehensive score, the threshold configuration, and the confidence information, are written into the similarity matrix as a complete audit basis for deduplication decisions and to provide an input basis for subsequent clustering and archiving.

[0055] Through the structured judgment mechanism based on the set threshold, the present invention achieves organic synergy between automation and manual intervention while ensuring high accuracy, significantly improving the practicality and robustness of large-scale document deduplication systems.

[0056] In summary, this implementation method constructs a hierarchical collaborative, multi-algorithm fusion process for rapid document deduplication: First, it standardizes the input format of heterogeneous documents from multiple sources through standardized preprocessing; then, it uses a Bloom filter to perform O(1) level coarse screening on the vocabulary set to quickly exclude absolutely non-duplicate documents; for documents that pass the coarse screening, it generates a 64-bit SimHash fingerprint based on TF-IDF vectors and a 128-dimensional MinHash signature based on the vocabulary set in parallel; it constructs a 16-band Global Local Sensitive Hash (LSH) index based on the SimHash fingerprint to achieve approximately O(1) candidate similar document retrieval; then, for only a very small candidate set, it calculates the similarity of the title (SimHash Hamming distance normalization), content (MinHash signature matching rate), and metadata (weighted or rule-based calculation after itemization) in three dimensions, and obtains a comprehensive similarity score by weighted fusion according to preset weights; finally, it achieves accurate determination of repetition, possible repetition, or non-repetition by comparing with a set threshold. This scheme effectively overcomes the limitations of a single algorithm, reducing the traditional O(n²) complexity to near-linear or even constant level, significantly improving processing efficiency. Simultaneously, through multi-dimensional feature fusion, it increases deduplication accuracy to over 95%, balancing high precision, high efficiency, and low resource consumption. Specifically, the standardized preprocessed deduplication vocabulary set is directly used as input to the Bloom filter, achieving efficient exclusion of absolutely non-repeating documents. Documents passing through the Bloom filter have their TF-IDF vectors and deduplication vocabulary set used in parallel to generate SimHash fingerprints and MinHash signatures, which serve different subsequent stages. The SimHash fingerprint is specifically used to construct a segmented LSH index to achieve near O(1) candidate set retrieval, while the MinHash signature is retained for content dimension similarity calculation in the actuarial stage. Finally, only a very small number of candidate sets selected by the LSH index are combined with pre-generated multi-dimensional features to perform weighted fusion and threshold determination of the three dimensions of similarity: title, content, and metadata. These steps constitute an inseparable overall technical solution, effectively solving the core problem of balancing efficiency and accuracy in large-scale document deduplication.

[0057] refer to Figure 3 This embodiment further discloses a fast document deduplication system based on multi-algorithm fusion, including a preprocessing module 21, a Bloom filter module 22, a multi-feature generation module 23, an index construction and retrieval module 24, a similarity calculation module 25, and a duplicate detection module 26, which are described in detail below: The preprocessing module 21 performs standardized preprocessing on the input document, extracting the title, body content, and metadata. Based on the body content, it generates a deduplicated vocabulary set to produce structured text features. The preprocessing module 21 receives the original document (supporting multiple formats such as TXT, PDF, DOC, DOCX, and HTML) and performs operations such as format parsing, text cleaning, word segmentation, stop word filtering, and TF-IDF vectorization, ultimately outputting a structured ProcessedDocument object. This object clearly encapsulates standardized data across three dimensions: title text, body content, and metadata, along with corresponding title vocabulary lists, body vocabulary lists, deduplicated vocabulary sets, and normalized TF-IDF vectors, providing a unified and high-quality input foundation for all subsequent algorithm modules.

[0058] Bloom filter module 22 is used to quickly determine the membership of the deduplication vocabulary set through a Bloom filter, excluding absolutely non-duplicate documents. Bloom filter module 22 maintains a global Bloom filter instance for quickly determining the membership of the document vocabulary set output by the preprocessing module. When a new document enters the system, this module checks whether all the corresponding bits of its words in the Bloom filter are 1: if any bit is 0, the document is determined to be "absolutely non-duplicate", and the subsequent process is terminated directly; otherwise, it is marked as "possibly existent" and handed over to the multi-feature generation module for processing. This module filters more than 90% of obviously non-duplicate documents in O(1) time complexity, significantly reducing the overall computational load of the system.

[0059] The multi-feature generation module 23 is used to generate, in parallel, a locally sensitive hash fingerprint based on semantic distribution and a probabilistic signature based on a deduplicated vocabulary set for documents that have passed through the Bloom filter. The multi-feature generation module 23 performs two feature calculation tasks in parallel: generating a 64-bit SimHash fingerprint based on the text TF-IDF vector output by the preprocessing module to capture the semantic distribution features of the document; and generating a MinHash signature based on the deduplicated vocabulary set using multiple independent hash functions to estimate the similarity of the vocabulary set. These two calculations are independent and can be executed in parallel. The output results are used in the LSH index construction and actuarial stages, respectively, achieving a balance between feature complementarity and computational efficiency.

[0060] The index building and retrieval module 24 is used to build a segmented local sensitive hash index based on the local sensitive hash fingerprint, and retrieve a set of candidate similar documents from the document library through the segmented local sensitive hash index.

[0061] The similarity calculation module 25 is used to calculate the similarity of the title dimension, content dimension and metadata dimension for each candidate similar document in the candidate similar document set, and to perform weighted fusion of the similarity of each dimension according to the preset weight to obtain the comprehensive similarity score of the document. The duplicate determination module 26 is used to compare the comprehensive similarity score with a set similarity threshold to determine whether the document is duplicated based on the comparison result.

[0062] Figure 4 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 4 As shown, the electronic device 50 includes: a processor 501, a memory 502, and a bus 503; The processor 501 and the memory 502 communicate with each other via the bus 503; the processor 501 is used to call the program instructions in the memory 502 to execute the methods provided in the above-described embodiments.

[0063] This embodiment provides a non-transitory computer-readable storage medium that stores computer instructions that cause a computer to execute the methods provided in the above-described embodiments.

[0064] Those skilled in the art will understand that all or part of the steps of the above-described method implementation can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above-described method implementation. The aforementioned storage medium includes various storage media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0065] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0066] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of each embodiment or some parts of the embodiments.

[0067] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A fast document deduplication method based on multi-algorithm fusion, characterized in that, Includes the following steps: Standardized preprocessing is performed on the input document to extract the title, body content and metadata, and a deduplicated vocabulary set is generated based on the body content to generate structured text features; The deduplication vocabulary set is quickly determined by a Bloom filter to exclude absolutely non-duplicate documents. For documents that pass through the Bloom filter, generate locality-sensitive hash fingerprints based on semantic distribution; A segmented local sensitive hash index is constructed based on the local sensitive hash fingerprint, and a set of candidate similar documents is retrieved from the document library using the segmented local sensitive hash index; For the set of candidate similar documents, the similarity of the title dimension, content dimension and metadata dimension is calculated for each candidate similar document, and the similarity of each dimension is weighted and fused according to the preset weight to obtain the comprehensive similarity score of the document. The overall similarity score is compared with a set similarity threshold to determine whether the documents are duplicates based on the comparison results.

2. The document deduplication method according to claim 1, characterized in that, The standardized preprocessing includes: The input document is parsed to extract the title text, body content, and metadata; The extracted title and body texts are cleaned to remove HTML tags, special characters, and non-standard whitespace characters. The cleaned title and body texts are segmented and stop words are filtered to generate a title vocabulary list, a body vocabulary list, and a deduplicated vocabulary set. Based on the title vocabulary list and the body vocabulary list, TF-IDF vectors are constructed and normalized to serve as structured text features.

3. The document deduplication method according to claim 2, characterized in that, The method further includes: generating a probabilistic signature based on the set of deduplicated words to which the document belongs while generating a locality-sensitive hash fingerprint; The Local Sensitive Hash fingerprint is a SimHash fingerprint, which generates a 64-bit binary fingerprint based on the TF-IDF vector. The probabilistic signature is a MinHash signature, which is based on the deduplicated vocabulary set and generates a signature vector composed of multiple MinHash values ​​through multiple independent hash functions.

4. The document deduplication method according to claim 1, characterized in that, The construction of the segmented locality-sensitive hash index based on the locality-sensitive hash fingerprint includes: The local sensitive hash fingerprint is divided into a preset number of bands, and each band contains a corresponding number of consecutive binary bits; For each band, calculate its band hash value and map the band hash value to the identifier of the corresponding document in the global inverted index; Based on the mapping relationship between the zoned hash values ​​of all documents and the document identifiers, the zoned local sensitive hash index is constructed.

5. The document deduplication method according to claim 3, characterized in that, The similarity in the title dimension is obtained by calculating and normalizing the Hamming distance between the SimHash title fingerprints of the target document and the candidate document; The content dimension similarity is obtained by comparing the number of times the hash values ​​at the same position in the MinHash signatures of the target document and the candidate document are equal, and then dividing the number by the total length of the MinHash signature; The similarity of the metadata dimension is obtained by calculating the similarity of each metadata item of the target document and the candidate document separately, and then taking a weighted average of the similarity of each metadata item or combining them according to a preset rule.

6. The document deduplication method according to claim 1 or 4, characterized in that, When retrieving a candidate similar document set using the segmented locality-sensitive hash index, each segmented hash value of the target document is used as a query key to find the corresponding document identifier list in the segmented locality-sensitive hash index, and a union operation is performed on all the retrieved document identifier lists to obtain the candidate similar document set.

7. A document deduplication system based on multi-algorithm fusion, characterized in that, include: The preprocessing module is used to perform standardized preprocessing on the input document, extract the title, body content and metadata, and generate a deduplicated vocabulary set based on the body content to generate structured text features; The Bloom filter module is used to quickly determine the membership of the deduplication vocabulary set and exclude absolutely non-duplicate documents; The multi-feature generation module is used to generate, in parallel, local sensitive hash fingerprints based on semantic distribution and probabilistic signatures based on deduplicated vocabulary sets from documents that have passed through the Bloom filter. The index building and retrieval module is used to build a segmented local sensitive hash index based on the local sensitive hash fingerprint, and retrieve a set of candidate similar documents from the document library through the segmented local sensitive hash index; The similarity calculation module is used to calculate the similarity of the title dimension, content dimension and metadata dimension for each candidate similar document in the candidate similar document set, and to perform weighted fusion of the similarity of each dimension according to the preset weight to obtain the comprehensive similarity score of the document. The duplicate determination module is used to compare the comprehensive similarity score with a set similarity threshold to determine whether the document is duplicated based on the comparison result.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1-6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method of any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-6.

Images