A method and system for detecting duplicate pages in power documents

Abstract

The application discloses a power document duplicate page detection method and system, and relates to the technical field of document detection. Through converting the processing mode of power documents based on OCR text analysis and semantic comparison into the feature fingerprint modeling and similarity grading judgment mode based on the page visual structure, the duplicate page recognition can be directly completed by using the overall layout structure of the page, the influence of high proportion of drawings, sparse text, scanning inclination, noise, watermark and seal shielding on the stability of OCR recognition and the accuracy of semantic matching is avoided, and through the combination of the visual feature fingerprint preliminary screening, Hamming distance filtering and structure similarity fine verification, the calculation complexity and recognition cost are reduced while the accuracy of duplicate page recognition is ensured, and the robustness and processing efficiency of duplicate page detection in the large-scale power engineering archive scene are improved.

Description

Technical Field

[0001] This invention relates to the field of document detection technology, specifically to a method and system for detecting duplicate pages in power documents. Background Technology

[0002] In the process of power engineering file management and operation and maintenance data archiving, a large number of construction drawings, survey records, equipment ledgers, signed confirmation pages, and various scanned documents are typically stored and managed in a unified manner. In actual business operations, these documents are often scanned multiple times, imported in batches, or archived across systems in electronic formats such as PDF or OFD. Due to factors such as different operators' processing habits, differences in scanning equipment resolution, and repeated submissions of documents during the circulation process, the same page or pages with highly similar structures frequently appear in different versions of documents, thus forming a large amount of potentially duplicate page data.

[0003] In existing technologies, methods based on OCR text recognition combined with text similarity comparison are commonly used to detect document duplication, or semantic matching is performed by extracting page text features to achieve duplicate page identification. However, in power engineering document scenarios, especially in scanned pages of construction drawings, topology diagrams, and tables, the page content is often dominated by graphic structures with a low proportion of text information and complex layout. Furthermore, issues such as scanning tilt, noise interference, watermark occlusion, and stamp coverage are common, leading to poor stability of OCR recognition results and easy text sequence disorder. Consequently, text-based duplication detection methods suffer from significant deficiencies in accuracy, robustness, and computational efficiency, making it difficult to meet the practical needs of efficient and stable automatic duplicate page identification in large-scale power engineering archives. Summary of the Invention

[0004] The purpose of this invention is to solve the problems mentioned in the background art, such as poor stability of OCR recognition results and easy disorder of text sequences, which makes the text-based duplicate detection method have obvious deficiencies in terms of accuracy, robustness and computational efficiency. Therefore, this invention proposes a method and system for detecting duplicate pages in power documents.

[0005] A first aspect of this invention provides a method for detecting duplicate pages in an electrical document, the method comprising: The system acquires the power documents to be tested, identifies the document format of the power documents, and determines the document parsing method based on the identification results. Convert each page in the power document into a corresponding page image, extract visual features from each page image, and generate a visual feature fingerprint that corresponds one-to-one with each page. Based on the Hamming distance between the visual feature fingerprints of different pages, all pages are initially screened to obtain candidate duplicate page pairs. Structural similarity is calculated for candidate duplicate pages to obtain fine similarity results between pages. Based on the fine similarity results, the duplicate page relationships are determined, and the duplicate page relationships are clustered to obtain the duplicate page grouping results and output them.

[0006] Optionally, the steps of converting each page in the power document into a corresponding page image, extracting visual features from each page image, and generating a visual feature fingerprint that corresponds one-to-one with each page are as follows: The process of generating page images includes image processing of PDF documents with resolution control, and the image resolution range is 150 to 200 DPI; The input page image is resized to adjust the original image to a preset size range. After completing the size normalization process, the scaled page image is converted into a grayscale image; Perform a discrete cosine transform on the grayscale image to map the image from the spatial domain to the frequency domain representation, and extract the low-frequency coefficients from it; After obtaining the low-frequency coefficients, they are subjected to statistical feature encoding. By discretizing the comparison between the low-frequency information and its mean or distribution characteristics, continuous numerical values ​​are mapped into binary judgment results. The discretization results are combined to generate a 64-bit binary perceptual hash value for the corresponding page, which serves as the visual feature fingerprint of the page image.

[0007] Optionally, the steps to obtain candidate duplicate pages are as follows: The 64-bit visual feature fingerprints corresponding to each page are stored uniformly, and an index relationship with the page number is established one-to-one. Based on the index relationship, the visual feature fingerprints of any two different pages are matched one by one, and the difference bit sequence is generated by XOR operation on the same position of the two 64-bit binary fingerprints. The Hamming distance between corresponding pages is obtained by counting the number of 1s in the difference bit sequence. If the Hamming distance value of a page is not greater than 5, the corresponding page will be marked as a candidate duplicate page and retained.

[0008] Optionally, based on the grayscale images corresponding to the two pages in the candidate duplicate page pair, the two page images are aligned in position according to the same size reference so that the two page images form a one-to-one corresponding comparison area at the same coordinate position; After the two page images have been aligned, they are synchronously divided into local windows of fixed size so that the two page images form corresponding first local regions and second local regions at the same position. For each pair of corresponding first local regions and second local regions, calculate the average gray value, gray standard deviation, and covariance between them. Substitute the brightness information, contrast information, and structure-related information of each corresponding local region into the structural similarity index calculation formula to obtain the local structural similarity value of the corresponding local window; The local structural similarity values ​​corresponding to all local windows are summarized to obtain the overall structural similarity index corresponding to the candidate duplicate pages, and the overall structural similarity index is used as the fine similarity result between pages.

[0009] Optionally, the formula for calculating the structural similarity index is:

[0010] in, This represents the structural similarity value of the i-th local window. This represents the average grayscale value of the local area of ​​the first page in the i-th local window. This represents the average grayscale value of the local area of ​​the second page in the i-th local window. This represents the standard deviation of grayscale values ​​in the local area of ​​the first page within the i-th local window. This represents the standard deviation of grayscale values ​​in the local area of ​​the second page within the i-th local window. This represents the covariance between the gray values ​​of two local regions within the i-th local window. and This represents a constant used to avoid a denominator of zero.

[0011] Optionally, the steps for determining page duplication relationships based on fine similarity results, clustering these relationships, and outputting the resulting groupings of duplicate pages are as follows: The fine similarity results for each candidate duplicate page are calculated, and the fine similarity results for each candidate duplicate page are compared with the preset duplicate judgment threshold. When the fine similarity result of a candidate duplicate page pair is not less than the preset duplicate judgment threshold, it is determined that there is a page duplication relationship between the two pages in the candidate duplicate page pair, and the corresponding page number pair is recorded. Construct a set of page duplication relationships based on all page IDs that have page duplication relationships, where each page ID is treated as an independent node, and an association is established between two page IDs that have page duplication relationships; Clustering is performed on the set of duplicate page relationships to group page numbers with direct or indirect duplicate relationships into the same page set, thus obtaining the duplicate page grouping results; Output the results of grouping duplicate pages, including the set of page numbers corresponding to each duplicate page group.

[0012] Optionally, the step of clustering the set of page repetition relationships includes: Initialize the page numbers with duplicate page relationships as independent nodes in the disjoint-setup, and use the page number itself as the initial representative node. A sequence of ordered repeating relationships is formed according to the page number order of the page number pairs, and the current representative node corresponding to each page number pair is read in turn; When two current representative nodes are different, calculate the minimum page number difference between the two corresponding page sets. When the minimum page number difference is not greater than the preset page number proximity threshold, take the page set with the smaller representative node page number value as the main set and take the page set with the larger representative node page number value as the secondary set. Then, redirect the representative node of the secondary set to the representative node of the main set to complete the set merging. After merging, the smallest page number in the merged result is used as the updated representative node; After traversing all page number pairs, the pages are categorized according to the final representative node corresponding to each page number, and page numbers with the same final representative node are grouped into the same duplicate page group.

[0013] A second aspect of the present invention provides a system for detecting duplicate pages in power documents, the system comprising: Parsing method module: acquires the power document to be detected, performs document format recognition on the power document, and determines the document parsing method based on the recognition result; Visual Feature Module: Converts each page in the power document into a corresponding page image, extracts visual features from each page image, and generates a visual feature fingerprint that corresponds one-to-one with each page. Candidate Duplicate Module: Based on the Hamming distance between the visual feature fingerprints of different pages, all pages are initially screened to obtain candidate duplicate page pairs; Detection module: Performs structural similarity calculation on candidate duplicate pages to obtain fine similarity results between pages, determines the duplicate relationship between pages based on the fine similarity results, performs clustering processing on the duplicate relationship between pages, obtains the duplicate page grouping results and outputs them.

[0014] The beneficial effects of this invention are: This invention proposes a method and system for detecting duplicate pages in power engineering documents. By transforming the traditional OCR-based text parsing and semantic comparison processing of power engineering documents into a feature fingerprint modeling and similarity classification judgment mechanism based on the visual structure of the page, the system can directly extract stable visual expressions from the overall layout structure of the page. It uses visual feature fingerprints to achieve low-cost and rapid screening of structural similarity between pages, and then combines Hamming distance for efficient filtering of candidate duplicate pages. Furthermore, it uses structural similarity index to conduct refined verification of candidate pages. This effectively avoids the problems of unstable OCR recognition, disordered text sequence, and semantic matching distortion caused by factors such as high proportion of drawings, sparse text information, scanning tilt, noise interference, watermarks, and seal obstruction in power engineering documents. While ensuring the accuracy of duplicate page recognition, it significantly reduces computational complexity and recognition cost, and improves the robustness and processing efficiency of duplicate page detection in large-scale power engineering archive scenarios. Attached Figure Description

[0015] Figure 1 This is a flowchart of a method for detecting duplicate pages in an electrical document, provided as an embodiment of the present invention. Detailed Implementation

[0016] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0017] This invention provides a method for detecting duplicate pages in power documents. See also... Figure 1 The method includes the following steps: S1: Obtain the power document to be detected, identify the document format of the power document, and determine the document parsing method based on the identification result; S2: Convert each page in the power document into a corresponding page image, extract visual features from each page image, and generate a visual feature fingerprint that corresponds to each page. S3: Based on the visual feature fingerprints of different pages, perform preliminary screening of all pages to obtain candidate duplicate page pairs; S4: Calculate the structural similarity of candidate duplicate pages to obtain the fine similarity results between pages. Based on the fine similarity results, determine the page duplication relationship and perform clustering on the page duplication relationship to obtain the duplicate page grouping results and output them.

[0018] In one embodiment, S1: Obtain the power document to be detected, identify the document format of the power document, and determine the document parsing method based on the identification result, specifically: Document format recognition includes PDF and OFD formats, with either a direct parsing method or an intermediate format conversion-based parsing method employed for OFD documents. Specifically, the power documents to be tested are first parsed and identified to determine whether they are in PDF or OFD format. For PDF documents, existing PDF parsing engines are used to parse each page and convert them into standard page images. For OFD documents, processing is tiered based on the complexity of their internal structure and parsing compatibility. When the OFD document structure is relatively standard, a lightweight direct parsing method is preferred to convert it into page images. When the OFD document contains vector graphics, complex topological drawings, or scanned nested structures, an intermediate format conversion path is used, i.e., the OFD is first converted to PDF and then further converted into page images. This ensures that OFD documents from different sources and with different complexities can be stably and completely processed into page-level images.

[0019] In one embodiment, S2: The steps of converting each page in the power document into a corresponding page image, extracting visual features from each page image, and generating a visual feature fingerprint that corresponds one-to-one with each page are as follows: The process of generating page images includes image processing of PDF documents with resolution control, and the image resolution range is 150 to 200 DPI; The input page image is scaled to adjust the original image to a preset scale range, which reduces computational complexity while preserving the overall layout and structural features of the page, so as to avoid structural distortion or feature shift caused by resolution differences. After size normalization, the scaled page image is converted into a grayscale image. By removing the influence of color information, the subsequent feature extraction process can focus more on the structural distribution and texture changes of the page, thereby improving the consistency and stability of page representation under different scanning conditions.

[0020] Performing Discrete Cosine Transform (DCT) on grayscale images maps the images from the spatial domain to the frequency domain and extracts the low-frequency coefficients. Since the low-frequency components mainly reflect the overall structural outline and layout features of the page, they can effectively characterize the global visual information of the page and have a strong ability to suppress local noise and high-frequency interference. After obtaining the low-frequency coefficients, they are subjected to statistical feature encoding. By discretizing the relationship between the low-frequency information and its mean or distribution characteristics, continuous numerical values ​​are mapped into binary judgment results, thereby forming a stable structural expression basis. The discretization results are combined to generate a 64-bit binary perceptual hash value for the corresponding page, which serves as the visual feature fingerprint of the page.

[0021] It should be noted that the above method for generating visual feature fingerprints corresponding to each page aims to balance structural stability and computational efficiency. Through progressive processing including size scaling, grayscale conversion, frequency domain transformation, and low-frequency information extraction, the page is gradually compressed from its original high-dimensional pixel space into low-dimensional stable features capable of representing the overall structure. Size scaling eliminates scale differences caused by varying scanning resolutions and unifies the structural representation; grayscale conversion removes non-structural interference such as color and stamps to enhance cross-device consistency; and DCT transformation converts spatial domain information into the frequency domain to achieve... The method allows for the separable representation of structural and noise information, while low-frequency coefficient extraction further preserves key global features such as page layout, frame structure, and overall typography, thereby avoiding interference from local noise in similarity judgment. Finally, continuous features are mapped to 64-bit binary perceptual hash values ​​through statistical encoding, enabling different pages to be represented in a compact and quickly comparable fingerprint format. Compared to methods such as direct comparison of original pixels, local feature point matching, or deep learning feature embedding, this method significantly reduces storage and computational complexity while ensuring strong robustness to scanning noise, slight deformation, and brightness changes.

[0022] In one embodiment, the step of initially screening all pages based on visual feature fingerprints of different pages to obtain candidate duplicate pages is as follows: The 64-bit visual feature fingerprints corresponding to each page are stored uniformly, and an index relationship with the page number is established to enable fast retrieval and matching. Based on the index relationship, the visual feature fingerprints of any two different pages are matched one by one, and the difference bit sequence is generated by XOR operation on the same position of the two 64-bit binary fingerprints. By counting the number of 1s in the difference bit sequence, the Hamming distance between the corresponding pages is obtained, which is used to characterize the degree of difference between the two pages at the visual structure level. If the Hamming distance value of a page is not greater than 5, the corresponding page will be marked as a candidate duplicate page and retained.

[0023] It should be noted that the 64-bit visual feature fingerprints generated for each page are first uniformly stored, and an index mapping relationship is established using the page number as a unique identifier. For example, the 64-bit binary fingerprints corresponding to pages 1, 2, and 3 are structured and stored to enable fast location and random access of fingerprint data. After the index is built, the system performs pairwise matching on all pages based on the index relationship. That is, any two different pages are selected as a page to be compared, and their corresponding 64-bit visual feature fingerprints are extracted from the index structure as input. Subsequently, a bitwise XOR operation is performed on the two 64-bit binary fingerprints at the same positions. If the corresponding bits are the same, the result is 0; if the corresponding bits are different, the result is 1. This generates a sequence of difference bits that reflects the distribution of structural differences. For example, when the fingerprint of page 1 matches the fingerprint of page 2, the result is 1. When two pages of fingerprints differ only in a few bits, their XOR result contains only a small number of 1s. After obtaining the sequence of differing bits, the bits with values ​​of 1 are statistically accumulated to obtain the Hamming distance value between the corresponding pages. This value is used to characterize the degree of difference between the two pages at the overall visual structure level. For example, when the Hamming distance of a pair of pages is 1 or 2, it indicates that the two pages are highly consistent in structural layout. When the Hamming distance increases significantly, it indicates that the structural differences are obvious. Finally, the calculated Hamming distance is compared with a preset similarity threshold (e.g., 5). When the Hamming distance is not greater than the threshold, the corresponding page is identified as a candidate duplicate page and retained for subsequent fine structural similarity calculation. When the Hamming distance is greater than the threshold, it is directly removed, thereby achieving efficient preliminary screening and computational compression of the entire set of page combinations.

[0024] The above method, which uses visual feature fingerprints for initial screening of all pages, is adopted because this screening process uses 64-bit perceptual hash fingerprints as a unified representation. By establishing a one-to-one index relationship between page numbers and fingerprints, the similarity judgment between different pages can be completed in a low-dimensional binary space, thus avoiding the high overhead and instability caused by direct computation in high-dimensional pixel space or complex structural feature space. Furthermore, by performing a bitwise XOR operation on the fingerprints of any two pages, the structural differences between pages are transformed into a countable binary difference sequence, which is further quantized using Hamming distance. This transforms the originally complex problem of image structural differences into simple bitwise operations. This approach addresses the counting problem, significantly reducing computational complexity. Simultaneously, by setting a Hamming distance threshold, it rapidly filters all page pairs, retaining only those with minimal structural differences as candidate duplicates. This enables the system to effectively eliminate the vast majority of obviously dissimilar page combinations in large-scale power document scenarios, avoiding computational redundancy and efficiency degradation caused by subsequent fine-grained structural similarity calculations across the entire space. Furthermore, it maintains high recall for potential duplicate pages while ensuring filtering speed. Therefore, compared to methods based on text features, local feature point matching, or deep learning-based full-scale similarity calculations, this approach offers lower computational overhead, clearer judgment rules, and stronger robustness against scanning noise.

[0025] In one embodiment, S4: Calculate the structural similarity of candidate duplicate pages to obtain fine similarity results between pages, determine the page duplication relationship based on the fine similarity results, perform clustering processing on the page duplication relationship, obtain the duplicate page grouping results, and output them.

[0026] In one implementation, the steps for calculating the structural similarity of candidate duplicate pages to obtain the fine-grained similarity results between pages are as follows: Based on the grayscale images of the two pages in the candidate duplicate page pair, the two page images are aligned according to the same size reference so that the two page images form a one-to-one corresponding comparison area at the same coordinate position. After the two page images have been aligned, they are synchronously divided into local windows of fixed size so that the two page images form corresponding first local regions and second local regions at the same position. For each pair of corresponding first local regions and second local regions, the average gray value, gray standard deviation, and covariance between them are calculated respectively. The average gray value is used to characterize brightness information, the gray standard deviation is used to characterize contrast information, and the covariance is used to characterize structure-related information. Substituting the brightness, contrast, and structure-related information of each corresponding local region into the Structural Similarity Index (SSIM) calculation formula, the local structural similarity value of the corresponding local window is obtained. The SSIM calculation formula is as follows:

[0027] in, This represents the structural similarity value of the i-th local window. This represents the average grayscale value of the local area of ​​the first page in the i-th local window. This represents the average grayscale value of the local area of ​​the second page in the i-th local window. This represents the standard deviation of grayscale values ​​in the local area of ​​the first page within the i-th local window. This represents the standard deviation of grayscale values ​​in the local area of ​​the second page within the i-th local window. This represents the covariance between the gray values ​​of two local regions within the i-th local window. and This represents a constant used to avoid a denominator of zero; The local structural similarity values ​​corresponding to all local windows are summarized to obtain the overall structural similarity index corresponding to the candidate duplicate pages, and the overall structural similarity index is used as the fine similarity result between pages.

[0028] It should be noted that, for example, if the first page is an original scanned page of a construction drawing, and the second page is a re-scanned page of the same drawing, with identical content but slight offsets and differences in brightness, the two page images are first aligned according to the same size reference, ensuring that the title bar, title frame, and table boundaries correspond on the coordinate plane. Then, the two aligned pages are simultaneously divided into fixed-size local windows, with each window compared only to windows in the same position on the other page. Next, the average gray value, gray standard deviation, and covariance are calculated for each pair of corresponding windows. The average gray value reflects the brightness level of the window, the gray standard deviation reflects the contrast strength of the window, and the covariance reflects the contrast between the two windows. The covariance of a window is determined by whether the grayscale variation trend within the window is consistent. For example, if a window is darker overall due to rescanning, but the distribution relationship between lines and background remains unchanged, its covariance is still relatively high. Then, the above parameters are substituted into the SSIM formula to obtain the local structural similarity value of the window. If the window content is consistent, the value is higher. If a window has a line structure on one page but is obscured by a stamp or is blank on another page, the value is lower. Finally, the structural similarity values ​​of all local windows are summarized to obtain the overall structural similarity index of the page. If the index is high, it means that although there are scanning differences between the two pages, their overall structure is consistent and they can be treated as duplicate pages. Otherwise, it means that the two pages are only similar in appearance but their content is different.

[0029] The advantage of using the above method to calculate the fine similarity between pages is that it does not simply compare the overall average of the entire page image, nor does it judge based solely on a single grayscale difference or contour difference. Instead, it first establishes a one-to-one correspondence of local comparison relationships under a unified size and position benchmark. Then, it uses the average grayscale value, grayscale standard deviation, and covariance to conduct a detailed analysis of each local area from three aspects: brightness, contrast, and structural correlation. This allows it to effectively distinguish between pages with the same content but different scanning conditions and pages that are slightly similar in appearance but have different actual content. For example, for two scanned versions of the same construction drawing, even if there are slight offsets, overall darkness, local noise, or black edges, as long as the title bar, drawing frame, circuit structure, and table layout are within the corresponding local area, they can still be compared. If the grayscale variation trend within the domain remains consistent, most local windows will still obtain high structural similarity values, and the overall structural similarity index will remain at a high level, thus avoiding misclassifying true duplicate pages as different pages. Conversely, for two pages that have similar visual fingerprints in the coarse screening stage, but one page is a parameter table and the other is a topology map, their corresponding local areas are difficult to keep consistent in terms of brightness distribution, contrast variation, and internal structural relationships. Therefore, low structural similarity values ​​will continue to be generated at the local window level, and the overall structural similarity index will also decrease significantly, thus avoiding misclassifying non-duplicate pages as duplicate pages. Therefore, this method not only improves the tolerance to scanning noise, brightness fluctuations, and slight deformations, but also enhances the ability to identify the true structural differences of pages.

[0030] In one embodiment, the steps of determining page duplication relationships based on fine similarity results, clustering these page duplication relationships, obtaining the duplicate page grouping results, and outputting them are as follows: The fine similarity results for each candidate duplicate page are calculated, and the fine similarity results for each candidate duplicate page are compared with the preset duplicate judgment threshold. When the fine similarity result of a candidate duplicate page pair is not less than the preset duplicate judgment threshold, it is determined that there is a page duplication relationship between the two pages in the candidate duplicate page pair, and the corresponding page number pair is recorded. Construct a set of page duplication relationships based on all page IDs that have page duplication relationships, where each page ID is treated as an independent node, and an association is established between two page IDs that have page duplication relationships; Clustering is performed on the set of duplicate page relationships to group page numbers with direct or indirect duplicate relationships into the same page set, thus obtaining the duplicate page grouping results; Output the results of grouping duplicate pages, including the set of page numbers corresponding to each duplicate page group.

[0031] It should be noted that after completing the SSIM fine similarity calculation, several candidate duplicate page pairs and their corresponding results are obtained. Assuming the fine similarity result for page pair (1,2) is 0.96, for page pair (2,5) it is 0.94, for page pair (8,9) it is 0.92, and for page pair (3,7) it is only 0.81, if the preset duplicate judgment threshold is set to 0.90, then the fine similarity result of each candidate page pair is first compared with this threshold. Page pairs (1,2), (2,5), and (8,9) are all considered duplicates due to... Pages with a value of 0.90 or higher are identified as having duplicate page relationships, and are recorded as duplicate associated page number pairs. Page pair (3,7) is not considered a duplicate relationship and is directly removed because it is below the threshold. Based on this, a set of page duplicate relationships is constructed based on all page number pairs that are confirmed to have duplicate relationships. Specifically, each page number is treated as an independent node. For example, pages 1, 2, 5, 8, and 9 are independent nodes. Then, a relationship is established between page 1 and page 2, a relationship is established between page 2 and page 5, and a relationship is established between page 8 and page 9. This establishes a set of relationships, forming several relational structures composed of nodes and connecting edges. Subsequently, clustering is performed on this set of repeated page relationships. The core of this clustering process is not to look at individual pages, but to identify whether there are indirect repeated relationships formed through intermediate pages. For example, if pages 1 and 2 are identified as repeated, and pages 2 and 5 are also identified as repeated, then although pages 1 and 5 may not have directly calculated fine-grained similarity results, they form a transitive relationship through page 2. Therefore, pages 1, 2, and 5 should be grouped into the same repeated page group. Pages 8 and 9, being only related to each other, form another unique group. Grouping is performed; finally, after merging all page relationships, the page numbers in each duplicate page group are output in set form. For example, the output results can be represented as [1,2,5] and [8,9], where the first group indicates that pages 1, 2, and 5 belong to the same duplicate page group, and the second group indicates that pages 8 and 9 belong to the same duplicate page group. This achieves a complete transformation from the similarity results of candidate page pairs to the duplicate page grouping results in the entire document, so that the final output results no longer remain at scattered pairs of pages, but form a grouped result that can be directly used for duplicate page marking, statistics, or cleaning.

[0032] In one implementation, the step of clustering the set of page duplication relationships to group page numbers with direct or indirect duplication relationships into the same page set includes: Initialize all page numbers with duplicate page relationships into independent nodes in the disjoint-set data structure, and use the page number itself as the initial representative node of the corresponding node. Sort the pages according to the first page number in ascending order. If the first page numbers are the same, sort them according to the second page number in ascending order to obtain an ordered repeating sequence. Read each pair of page numbers sequentially according to the ordered repeating relationship sequence, and query the current representative node corresponding to the two page numbers in the pair of page numbers respectively; When the current representative nodes corresponding to two page numbers are different, the set merging is not performed directly. Instead, the minimum page number difference in the page sets corresponding to the two representative nodes is calculated first. The minimum page number difference is the minimum absolute value of the difference between any page number in one page set and any page number in another page set. When the minimum page number difference is not greater than the preset page number proximity threshold, it is determined that the two page sets meet the sequential merging condition. The page set with the smaller representative node page number value is taken as the main set, and the page set with the larger representative node page number value is taken as the slave set. Then, the representative node of the slave set is redirected to the representative node of the main set to complete the merging of the two page sets. After the set merging is completed, all page numbers in the main set and all page numbers in the secondary set are uniformly registered, and the smallest page number in the merged set is used as the updated representative node. The correspondence between the updated representative node and its page number is rewritten. When the current representative node corresponding to two page numbers is the same, keep the original set affiliation unchanged and continue reading the next set of page number pairs; When the minimum page number difference is greater than the preset page number proximity threshold, the two page sets remain independent of each other, no merging operation is performed, and the next set of page number pairs is read. After traversing all page numbers, the pages are categorized according to their final representative nodes. All page numbers with the same final representative node are grouped into the same duplicate page group to obtain the duplicate page grouping result.

[0033] It should be noted that the above method is used to cluster the set of repeated page relationships, rather than directly using ordinary graph clustering, ordinary connected component merging, or unconstrained merging methods such as regular union. This is because, although the repeated page relationships are derived from fine-grained similarity determination, in the context of power documents, there may still be situations where pages are connectable but should not be directly merged due to similar local structures, identical cross-chapter templates, or similar page layouts in different locations. Therefore, if set merging is performed directly based solely on the existence of repeated edges, it is easy to cause large-scale mis-clustering triggered by a small number of similar pages. This solution first initializes all page numbers during the clustering process. Initialized as independent nodes, and then forming an ordered repeating sequence according to the page number order, the entire merging process proceeds step by step following the page number expansion order within the document, avoiding set drift caused by disordered merging from the source. Subsequently, when merging two page sets each time, they are not merged immediately, but the minimum page number difference between the two sets is calculated first, and sequential merging is only allowed if the minimum page number difference is not greater than a preset page number proximity threshold. This means that set merging not only requires a repeating relationship, but also requires adjacent or close characteristics in page number distribution, thus preventing duplicate pages in power documents from typically originating from continuous scanning and repeated insertions. The actual distribution patterns, such as those of inbound and partial batch archiving, are directly incorporated into the clustering criteria. This effectively suppresses the risk of cross-chapter, cross-region, and cross-template pages being mistakenly grouped into the same duplicate page group due to individual similarities. Furthermore, after meeting the merging conditions, this scheme further limits the set of pages with smaller representative node page numbers to the main set and the set of pages with larger page numbers to the secondary set. After merging, the smallest page number in the merged result is uniformly used as the updated representative node. This ensures that each duplicate page group always maintains a stable sequential anchor point, avoiding the problems of ordinary disjoint-set data structures where representative nodes are affected by path compression or random parent node selection. The above method addresses issues such as instability and poor traceability of grouping results. Therefore, its advantage lies in not simply pursuing clustering speed, but rather maintaining the efficient merging capability of the disjoint-set data structure. Through a specific mechanism—page number order-driven, proximity threshold-limited, master-slave set merging, and rewriting the smallest page number representative node—the final grouping results of duplicate pages better reflect the actual formation patterns of duplicate pages in power documents. This reduces the probability of mis-clustering, improves the stability of grouping boundaries and the consistency of results, and makes the formation path of each group of duplicate pages clearer and more interpretable, facilitating subsequent retention, deletion, annotation, and manual review of duplicate pages.

[0034] Based on the same inventive concept, embodiments of the present invention also provide a system for detecting duplicate pages in power documents. This system includes: Parsing method module: acquires the power document to be detected, performs document format recognition on the power document, and determines the document parsing method based on the recognition result; Visual Feature Module: Converts each page in the power document into a corresponding page image, extracts visual features from each page image, and generates a visual feature fingerprint that corresponds one-to-one with each page. Candidate Duplicate Module: Based on the Hamming distance between the visual feature fingerprints of different pages, all pages are initially screened to obtain candidate duplicate page pairs; Detection module: Performs structural similarity calculation on candidate duplicate pages to obtain fine similarity results between pages, determines the duplicate relationship between pages based on the fine similarity results, performs clustering processing on the duplicate relationship between pages, obtains the duplicate page grouping results and outputs them.

[0035] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should still fall within the scope of the claims of the present invention.

Claims

1. A method for detecting duplicate pages in power documents, characterized in that, Includes the following steps: The system acquires the power documents to be tested, identifies the document format of the power documents, and determines the document parsing method based on the identification results. Convert each page in the power document into a corresponding page image, extract visual features from each page image, and generate a visual feature fingerprint that corresponds one-to-one with each page. Based on the Hamming distance between the visual feature fingerprints of different pages, all pages are initially screened to obtain candidate duplicate page pairs. Structural similarity is calculated for candidate duplicate pages to obtain fine similarity results between pages. Based on the fine similarity results, the duplicate page relationships are determined, and the duplicate page relationships are clustered to obtain the duplicate page grouping results and output them.

2. The method for detecting duplicate pages in an electrical document according to claim 1, characterized in that, The steps to convert each page in the power document into a corresponding page image, extract visual features from each page image, and generate a visual feature fingerprint that corresponds one-to-one with each page are as follows: The process of generating page images includes image processing of PDF documents with resolution control, and the image resolution range is 150 to 200 DPI; The input page image is resized to adjust the original image to a preset size range. After completing the size normalization process, the scaled page image is converted into a grayscale image; Perform a discrete cosine transform on the grayscale image to map the image from the spatial domain to the frequency domain representation, and extract the low-frequency coefficients from it; After obtaining the low-frequency coefficients, they are subjected to statistical feature encoding. By discretizing the comparison between the low-frequency information and its mean or distribution characteristics, continuous numerical values ​​are mapped into binary judgment results. The discretization results are combined to generate a 64-bit binary perceptual hash value for the corresponding page, which serves as the visual feature fingerprint of the page image.

3. The method for detecting duplicate pages in an electrical document according to claim 1, characterized in that, The steps to obtain candidate duplicate page pairs are as follows: The 64-bit visual feature fingerprints corresponding to each page are stored uniformly, and an index relationship with the page number is established one-to-one. Based on the index relationship, the visual feature fingerprints of any two different pages are matched one by one, and the difference bit sequence is generated by XOR operation on the same position of the two 64-bit binary fingerprints. The Hamming distance between corresponding pages is obtained by counting the number of 1s in the difference bit sequence. If the Hamming distance value of a page is not greater than 5, the corresponding page will be marked as a candidate duplicate page and retained.

4. The method for detecting duplicate pages in an electrical document according to claim 1, characterized in that, Based on the grayscale images of two pages in the candidate duplicate pages, the two page images are aligned according to the same size reference so that the two page images form a one-to-one corresponding comparison area at the same coordinate position. After the two page images have been aligned, they are synchronously divided into local windows of fixed size so that the two page images form corresponding first local regions and second local regions at the same position. For each pair of corresponding first local regions and second local regions, calculate the average gray value, gray standard deviation, and covariance between them. Substitute the brightness information, contrast information, and structure-related information of each corresponding local region into the structural similarity index calculation formula to obtain the local structural similarity value of the corresponding local window; The local structural similarity values ​​corresponding to all local windows are summarized to obtain the overall structural similarity index corresponding to the candidate duplicate pages, and the overall structural similarity index is used as the fine similarity result between pages.

5. The method for detecting duplicate pages in an electrical document according to claim 4, characterized in that, The formula for calculating the structural similarity index is as follows: in, This represents the structural similarity value of the i-th local window. This represents the average grayscale value of the local area of ​​the first page in the i-th local window. This represents the average grayscale value of the local area of ​​the second page in the i-th local window. This represents the standard deviation of grayscale values ​​in the local area of ​​the first page within the i-th local window. This represents the standard deviation of grayscale values ​​in the local area of ​​the second page within the i-th local window. This represents the covariance between the gray values ​​of two local regions within the i-th local window. and This represents a constant used to avoid a denominator of zero.

6. The method for detecting duplicate pages in an electrical document according to claim 1, characterized in that, The steps for determining page duplication relationships based on fine similarity results, clustering these relationships, and outputting the resulting grouped duplicate pages are as follows: The fine similarity results for each candidate duplicate page are calculated, and the fine similarity results for each candidate duplicate page are compared with the preset duplicate judgment threshold. When the fine similarity result of a candidate duplicate page pair is not less than the preset duplicate judgment threshold, it is determined that there is a page duplication relationship between the two pages in the candidate duplicate page pair, and the corresponding page number pair is recorded. Construct a set of page duplication relationships based on all page IDs that have page duplication relationships, where each page ID is treated as an independent node, and an association is established between two page IDs that have page duplication relationships; Clustering is performed on the set of duplicate page relationships to group page numbers with direct or indirect duplicate relationships into the same page set, thus obtaining the duplicate page grouping results; Output the results of grouping duplicate pages, including the set of page numbers corresponding to each duplicate page group.

7. The method for detecting duplicate pages in an electrical document according to claim 6, characterized in that, The steps for clustering a set of duplicate page relationships include: Initialize the page numbers with duplicate page relationships as independent nodes in the disjoint-setup, and use the page number itself as the initial representative node. A sequence of ordered repeating relationships is formed according to the page number order of the page number pairs, and the value of the current representative node corresponding to each page number pair is read in sequence. When two current representative nodes are different, calculate the minimum page number difference between the two corresponding page sets. When the minimum page number difference is not greater than the preset page number proximity threshold, take the page set with the smaller representative node page number value as the main set and take the page set with the larger representative node page number value as the secondary set. Then, redirect the representative node of the secondary set to the representative node of the main set to complete the set merging. After merging, the smallest page number in the merged result is used as the updated representative node; After traversing all page number pairs, the pages are categorized according to the final representative node corresponding to each page number, and page numbers with the same final representative node are grouped into the same duplicate page group.

8. A system for detecting duplicate pages in power documents, used to implement the method for detecting duplicate pages in power documents according to any one of claims 1-7, characterized in that, The system includes: Parsing method module: acquires the power document to be detected, performs document format recognition on the power document, and determines the document parsing method based on the recognition result; Visual Feature Module: Converts each page in the power document into a corresponding page image, extracts visual features from each page image, and generates a visual feature fingerprint that corresponds one-to-one with each page. Candidate Duplicate Module: Based on the Hamming distance between the visual feature fingerprints of different pages, all pages are initially screened to obtain candidate duplicate page pairs; Detection module: Performs structural similarity calculation on candidate duplicate pages to obtain fine similarity results between pages, determines the duplicate relationship between pages based on the fine similarity results, performs clustering processing on the duplicate relationship between pages, obtains the duplicate page grouping results and outputs them.

Images