Financial document duplicate removal method and system based on Hash and rule self-learning
By employing a hash- and rule-based self-learning method, non-content-related internal structural data in financial announcement PDFs is removed. A two-stage deduplication process using SHA-256 and fuzzy hash algorithms is implemented, solving the problems of high false positive and high false negative rates in existing technologies and achieving efficient and accurate deduplication of financial documents.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HEFEI DAZHIHUI CAIHUI DATA TECH CO LTD
- Filing Date
- 2025-12-23
- Publication Date
- 2026-04-17
AI Technical Summary
Existing technologies suffer from high false positive and false negative rates in the deduplication process of financial announcement PDF files, and cannot achieve accurate and automated deduplication. This is mainly due to the reliance on external metadata leading to content misjudgment and the reliance on non-content-related internal structure data leading to unstable hash values.
A hash- and rule-based self-learning approach is adopted. Non-content-based internal structure data is removed through a noise rule base. First-level deduplication is performed using SHA-256 hash values, and second-level deduplication is performed by combining fuzzy hashing algorithm. The self-learning mechanism updates the noise rule base to adapt to noise patterns from different sources.
It significantly improves the accuracy of deduplication and the system's adaptability, reduces the consumption of computing resources, and achieves millisecond-level fast response and high throughput processing, making it suitable for real-time deduplication scenarios involving massive amounts of financial documents.
Smart Images
Figure CN121882016A_ABST
Abstract
Claims
1. A method for deduplicating financial documents based on hashing and rule self-learning, characterized in that, include: S1: Obtain the financial PDF document and the corresponding document's complete byte stream, and encode them to obtain the document encoding; S2: The document encoding is processed using regular expressions based on rules in the noise rule base to remove noise and obtain standardized pipeline data of the document; S3: Perform first-level deduplication by comparing all documents using SHA-256 hash values; S4: For the retained documents, perform similarity filtering by comparing metadata, and then use the fuzzy hash algorithm to perform secondary deduplication, thereby obtaining the deduplicated documents.
2. The method for deduplicating financial documents based on hashing and rule self-learning as described in claim 1, characterized in that, In the second-level deduplication process, before deleting documents identified as duplicates, the rules in the noise rule base are added and updated based on the duplicate documents. When the number of rules in the noise rule base reaches the set rule threshold and the number of documents with similarity greater than the similarity threshold in the second-level deduplication process is less than the set document number threshold, subsequent document deduplication is performed directly in the first-level deduplication process, without the need for the second-level deduplication.
3. The method for deduplicating financial documents based on hashing and rule self-learning as described in claim 1, characterized in that, Step S1 includes: Obtain the full byte stream of a financial PDF document : in, Indicates the path to the PDF document file; Function to obtain the complete byte stream of a file; Then obtain the complete byte stream The document encoding is uniformly converted to UTF-8 format. .
4. The method for deduplicating financial documents based on hashing and rule self-learning as described in claim 1, characterized in that, In step S2, the document's standardized pipeline data Obtain it through the following methods: in, This represents the noise rule base; n is the total number of rules. Encode the document; (.) represents the noise removal function, which uses regular expressions to remove noise from the document encoding based on rules in a noise rule base. Used for noise removal.
5. The method for deduplicating financial documents based on hashing and rule self-learning as described in claim 1, characterized in that, The rules in the noise rule base include whitespace characters, timestamp characters, source characters, document identifiers, and font hash values.
6. The method for deduplicating financial documents based on hashing and rule self-learning as described in claim 1, characterized in that, Step S3 includes: First, based on the standardized pipeline data BSD for each document, calculate the corresponding SHA-256 hash value. The calculation method is as follows: in, (.) is the function for obtaining the SHA-256 hash value; Then, the SHA-256 hash value of each document is compared with all other documents; if they are all different, the corresponding document is retained; if they are the same, the documents with the same SHA-256 hash value are determined to be duplicate documents, and the duplicate documents are retained or discarded according to the predetermined metadata priority rules.
7. The method for deduplicating financial documents based on hashing and rule self-learning as described in claim 1, characterized in that, Step S4 includes: S41: For the retained documents, group them according to the set conditions, and group the documents that meet the set conditions into a group, thereby obtaining several groups of documents; The conditions are set as follows: The document file size difference is less than the file difference threshold; the creation time window difference is less than the creation time difference threshold; and the title similarity is greater than the similarity threshold. The title similarity is calculated as follows: Where a and b are any two documents in the retained list; edidistance(.) is the edit distance, which is the minimum number of single-character edit operations required to convert one string into another; title(.) is the title; len(.) is the length; S42: For each group of documents, calculate the similarity between the documents using the following formula: The TLSH_distance(.) function is used to calculate the distance between the TLSH hashes of two documents. This is the maximum distance value; (.) is the similarity hash function used to calculate the TLSH hash value. , These represent the standardized pipeline data for any two documents, a and b, in each document group. S43: Determine whether the similarity between documents is greater than the similarity threshold; if yes, it is determined to be a duplicate document, and duplicate documents are retained or discarded according to the predetermined metadata priority rules; if no, the corresponding document is retained and stored.
8. The method for deduplicating financial documents based on hashing and rule self-learning as described in claim 2, characterized in that, The rules in the noise rule base are added and updated based on the documents that are identified as duplicates, including: S201: For each document identified as duplicate in the second-level deduplication process, the content-defined segmentation algorithm is used to segment the document into blocks, resulting in ordered blocks for each document; S202: Extract differences from ordered blocks of duplicate documents to obtain a set of difference blocks for duplicate documents. The extraction formula is: in, and These are ordered blocks obtained from documents a and b using a content-defined block segmentation algorithm, respectively. This is a symmetric difference operation, which removes the duplicate parts and leaves the non-duplicate parts to form a difference block pair; i refers to the number of the difference block, that is, the number of the byte blocks that differ between the two documents; The i-th difference block pair refers to the difference block pair formed by the i-th difference block in the two documents; The set of all differing block pairs in two documents; S203: Convert the set of difference blocks into rules and store them in the noise rule base. Add and update the rules in the noise rule base.
9. A method for deduplicating financial documents based on hashing and rule self-learning as described in claim 8, characterized in that, In step S203, the conversion of the difference block set into rules includes: First, for each pair of differing blocks, the longest common subsequence of the two differing blocks is calculated using a dynamic programming-based longest common subsequence extraction method; Then, by backtracking, the sequence of consecutive strings with differences is located and extracted to obtain a set of differences. Then, for each pair of difference content in the difference content set, pattern recognition is performed to generalize the specific difference value into a rule in the noise rule base.
10. A financial document deduplication system based on hashing and rule self-learning, characterized in that, To implement a financial document deduplication method based on hash and rule self-learning as described in any one of claims 1-9, the method includes a document acquisition module, a noise removal module, a first-level deduplication module, and a second-level deduplication module; the document acquisition module is used to acquire financial PDF documents and the complete byte stream of the corresponding documents and encode them; the noise removal module is used to denoise the encoded documents; the first-level deduplication module is used to deduplicatize the denoised documents using SHA-256 hash values; and the second-level deduplication module is used to further deduplicatize the documents retained in the first-level deduplication.