A PDF file preprocessing method

Abstract

This invention discloses a PDF file preprocessing method, belonging to the field of document processing technology. It aims to solve the problems of mixed layout recognition and correction, seal interference removal, and large PDF file classification in existing PDF preprocessing tools, thereby addressing issues such as content loss, recognition errors, and low efficiency in reading and recognizing scanned / copied PDFs. This method determines the type of the PDF file and specifically addresses mixed layout, seal interference, and large file classification issues, sequentially completing layout correction, seal removal, and file classification, ultimately outputting a preprocessed PDF file. This invention is applicable to machine-generated PDFs, traditional scanned PDFs, and scanned PDFs with OCR, and is particularly effective in efficiently processing complex PDFs with mixed layouts, seals, and large file sizes, significantly improving the accuracy and efficiency of subsequent content recognition and providing a high-quality document data foundation for enterprise AI knowledge base construction and RAG technology applications.

Description

Technical Field

[0001] This invention relates to the field of document processing technology, specifically to a PDF file preprocessing method applicable to scenarios such as enterprise AI knowledge base construction, RAG (Retrieval Enhanced Generation) technology application, and document digitization management. It is particularly effective for complex PDF files with mixed layouts, stamp interference, and large file sizes, improving the accuracy and efficiency of subsequent content recognition and analysis through preprocessing. Background Technology

[0002] With the rapid development of large-scale models and RAG technology, the construction of enterprise AI knowledge bases needs to be based on high-quality material analysis. PDF files, as the primary carrier of enterprise documents (such as contracts, invoices, scanned copies of ID cards, business records, etc.), directly impact the application value of the knowledge base through their analysis quality. Currently, common PDF files are mainly divided into three categories: machine-generated PDFs (editable and searchable), traditional scanned PDFs (image collections, not editable), and scanned PDFs with OCR (containing searchable text layers, but with recognition errors). Traditional scanned PDFs and scanned PDFs with OCR account for over 60% of enterprise documents, representing the key focus and challenge in preprocessing.

[0003] Existing PDF processing tools (such as the open-source PaddleOCR, the commercial Baidu OCR, and Alibaba Cloud OCR) have many shortcomings in practical applications and are difficult to meet the needs of complex scenarios:

[0004] Insufficient ability to handle mixed layouts: Most tools only determine the layout direction by judging the Rotate attribute of the first page of the PDF. For mixed layout files that contain horizontal, vertical and inverted layouts, content loss or misidentification is likely to occur.

[0005] Stamp interference leads to recognition errors: Existing technologies mostly use standard OCR recognition methods, such as color space conversion, noise processing, edge detection, text edge protection, and related image restoration algorithms. These methods are more suitable for OCR scenarios where content recognition requirements are not high. Removing stamps in this way can easily damage the text content, especially in densely populated areas of tables (small text and thin lines). After removing the stamp, table lines are prone to breakage, text blurring, or loss.

[0006] Large file classification is inefficient: For large mixed-type files of hundreds of pages and hundreds of megabytes (such as bank-integrated user loan records, contracts and identity documents), existing tools (such as Microsoft Azure OCR and Tencent Cloud OCR) either time out or cannot process them, or can only recognize them page by page but cannot classify and generate sub-files.

[0007] For example, the patent with publication number CN117852499A discloses a method and apparatus for preprocessing PDF documents. This patent performs layout analysis and region segmentation on the PDF and reorganizes paragraphs based on attribute information, but does not involve key preprocessing steps such as PDF type judgment, stamp interference processing, and large file classification, and its functional dimensions are limited.

[0008] For example, the publication (announcement) number CN110807309A discloses a method, device and electronic device for recognizing the content type of PDF documents. This patent relies on training data and model performance, and its applicability is limited to the training set. For special or complex PDF formats outside the training data, the recognition effect may decrease. Moreover, the function is single, only solving the problem of type recognition, and does not form a complete processing flow including correction, classification and other functions.

[0009] For example, the publication (announcement) number CN118799871A discloses an electronic seal orientation correction system, method, device, and storage medium. This scheme uses a matting algorithm to extract the seal and corrects the orientation of the seal image through contour detection and bounding box calculation. It aims to correct the orientation of the seal itself, rather than removing the seal's interference with the background text. It only processes the seal orientation and is not integrated into a broader PDF preprocessing workflow.

[0010] Furthermore, existing tools suffer from significant cost variations in processing thousands of pages (commercial tools range from ¥20 to ¥180) and low efficiency in recognizing mixed-type documents. Scanned documents (images) often contain extremely fine text and table lines obscured by stamps, leading to extraction failures or errors with current methods, or requiring expensive machine learning approaches. Therefore, there is an urgent need for a PDF preprocessing method that can simultaneously address mixed layout correction, efficient stamp removal, and intelligent classification of large documents, resolving the pain points of existing technologies and providing high-quality data for subsequent document parsing. Summary of the Invention

[0011] The purpose of this invention is to address the shortcomings of existing technologies by proposing a PDF file preprocessing method.

[0012] To achieve the above objectives, the present invention adopts the following technical solution:

[0013] A PDF file preprocessing method specifically includes the following steps:

[0014] Step 1: PDF Type Determination: Obtain the PDF file to be processed and determine the type of the PDF file;

[0015] Step 2, Mixed Layout Processing: For each page of the PDF file, the layout direction is determined and corrected based on the relevant layout attributes.

[0016] Step 3: Seal Interference Handling: For PDF file pages containing seals, the seals are removed and the text content is protected through color feature separation and algorithm optimization.

[0017] Step 4: Large file classification and processing: For files of different sizes or number of pages, especially large PDF files, extract page feature information, classify the files according to the feature information, and generate sub-files of the corresponding types;

[0018] Step 5: Output preprocessing results: Integrate the processing results from Steps 2 to 4, and output the preprocessed PDF file.

[0019] Preferably, the PDF file type includes machine-generated PDF, traditional scanned PDF, and scanned PDF with OCR; the type is determined by parsing the PDF file metadata, judging the image ratio, and the amount of text.

[0020] Preferably, in step two, the layout-related attributes include the Rotate attribute of the PDF page; during correction, the Rotate attribute of each page is determined separately, and each page is adjusted individually to the correct layout angle.

[0021] Preferably, in step three, the color feature separation includes the core idea of ​​removing the stamp (based on the main color detection, applicable to stamps of any color).

[0022] Preferably, in step three, the seal is removed based on the assumption of black text on a white background:

[0023] Document text: Black / Gray → All three channel values ​​are low (R / G / B are not high)

[0024] Paper background: White / light color → All three channel values ​​are relatively high, but relatively balanced (R≈G≈B)

[0025] Seals usually have obvious color deviations, for example:

[0026] Red stamp: R channel is much higher than G and B.

[0027] Blue badge: Channel B is much higher than channels R and G.

[0028] Green badge: G channel is much higher than R and B.

[0029] The algorithm determines whether a pixel is the color of the stamp by combining two types of conditions:

[0030] 1. Absolute brightness threshold: A channel value > channel_threshold or > channel_high

[0031] 2. Color ratio threshold: The ratio of this channel to other channels > ratio_threshold

[0032] If the channel value is high enough and the color bias is obvious, it is identified as a stamp pixel and then whitened.

[0033] By using dual detection of channel absolute value and color ratio, it avoids the complex processing of conventional color space conversion, morphological optimization (noise processing), contour detection and analysis, edge protection, and image restoration. While improving performance by 10 times or more, the accuracy of extracting text covered by stamps is also close to 100%, making it particularly suitable for extracting text from scanned documents such as tables and PDFs with particularly fine table lines.

[0034] Preferably, in step four, PDFs that meet different size or page number conditions are processed (generally, PDFs with a file size ≥ 100MB or a page count ≥ 100 pages cannot be processed). For PDFs of different sizes, the page feature information includes the title information on the page. The title information must meet at least one of the following characteristics: located in the header area of ​​the page, font size larger than the body text size of the page, bold font style, no punctuation marks, and character length ≤ 20 characters to extract type features.

[0035] Preferably, in step four, when extracting title information, the title in the text layer of the machine-generated PDF is read directly (based on logical judgments such as position and text size); when classifying, a set of file types to be classified can be pre-input, and the classification is completed based on the matching degree between the title information and the set of types, with a classification accuracy of ≥90%.

[0036] Preferably, in step four, when generating sub-files, the page range of the original PDF file corresponding to each sub-file is recorded, and the format of the sub-file is consistent with the format of the original PDF file.

[0037] Furthermore, the algorithm for removing the stamp in step 3 is implemented using a pixel-level joint thresholding model, specifically including the following sub-steps:

[0038] I. Symbol Definition: Let the input color image be I. For any pixel position p, its BGR three-channel value is Define the mean of non-target channels:

[0039]

[0040]

[0041]

[0042] Define channel advantage ratio:

[0043] II. Threshold Parameter Settings: Set the channel absolute intensity threshold T, channel dominance ratio threshold K, and channel highlight threshold H;

[0044] III. Seal Pixel Discrimination: The red, blue, and green seal pixels are distinguished separately. The discrimination formula for the red seal pixels is:

[0045]

[0046] The discriminant for the blue stamp pixels is:

[0047]

[0048] The discriminant for the green stamp pixels is:

[0049]

[0050] The seal mask is obtained by taking the union of the three discrimination results.

[0051] IV. Mask Replacement and Background Fill: Perform background filling on pixels identified as stamps, and output the image:

[0052]

[0053] Furthermore, the channel absolute intensity threshold T corresponds to channel_threshold, the channel highlight threshold H corresponds to channel_high, and the channel advantage ratio threshold K corresponds to ratio_threshold;

[0054] In the process of determining the stamp pixels, when the mean value of the non-target channels is 0, the channel dominance ratio is set to 0 or a conditional division is performed according to the engineering implementation.

[0055] The algorithm is a pixel-by-pixel constant-time operation with an overall complexity of O(N), where N is the number of pixels in the image.

[0056] The present invention provides a PDF file preprocessing method, which, compared with the prior art, has the following advantages:

[0057] Improved processing accuracy: Mixed layout correction accuracy ≥98%, stamp residue rate ≤1%, text clarity loss rate ≤5%, providing high-quality images for subsequent OCR recognition, improving recognition accuracy by 30%-50%, and avoiding business data errors caused by layout or stamp issues;

[0058] Improved processing efficiency: The processing time for large file classification has been reduced from more than 10 minutes to 1 second. The processing speed for mixed typesetting and stamp removal is 10 times faster than existing professional methods, meeting the efficiency needs of enterprises for batch processing (such as banks processing thousands of pages of documents every day).

[0059] Reduced processing costs: Based on an open-source technology framework, there are no additional hardware costs. The processing cost per thousand pages is less than 1 / 9 of that of commercial tools (such as Tencent Cloud OCR ¥180 / thousand pages), while avoiding repeated processing costs caused by recognition errors.

[0060] Expanding applicable scenarios: Applicable to three PDF types: machine-generated, traditional scanning, and scanning with OCR, covering mainstream documents such as enterprise contracts, invoices, and identity documents, providing reliable data preprocessing support for AI knowledge base construction and RAG technology implementation, and helping enterprises with digital transformation. Attached Figure Description

[0061] Figure 1 Here is a flowchart of the preprocessing method for this PDF file;

[0062] Figure 2 A PDF file image before the seal processing;

[0063] Figure 3 This is an image of a PDF file after stamp processing based on this PDF file preprocessing method;

[0064] Figure 4 This is a schematic diagram illustrating the process of obtaining a PDF file to be processed and determining the type of the PDF file based on this PDF file preprocessing method.

[0065] Figure 5 This is a schematic diagram of hybrid typesetting processing based on this PDF file preprocessing method;

[0066] Figure 6 This is a schematic diagram illustrating the classification and processing of large files based on this PDF file preprocessing method;

[0067] Figure 7 This diagram illustrates how the PDF file preprocessing method described in this paper achieves automated classification through "title extraction - intelligent classification - sub-file generation". Figure 1 ;

[0068] Figure 8 This diagram illustrates how the PDF file preprocessing method described in this paper achieves automated classification through "title extraction - intelligent classification - sub-file generation". Figure 2 . Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0070] Example 1

[0071] like Figures 1 to 8 As shown, a PDF file preprocessing method includes five main steps: PDF type determination, mixed layout processing, stamp interference handling, large file classification processing, and outputting preprocessing results. The specific operations of each step are as follows:

[0072] Step 1: PDF type determination

[0073] Obtain the PDF file to be processed and determine the type of the PDF file;

[0074] Type Classification: PDF files are clearly divided into three categories: machine-generated PDFs, traditional scanned PDFs, and scanned PDFs with OCR;

[0075] Judgment criteria: Accurate classification is achieved through dual detection.

[0076] Count the number of image elements (LTFigure) on the page and calculate the proportion of image elements to all elements.

[0077] Extract all extractable text content from the page and determine the number of texts.

[0078] This dual-judgment method can effectively avoid misjudgment; for example, a PDF page containing many illustrations but still having normal text will not be mistaken for a scanned document.

[0079] This step allows for quick differentiation of PDF types. For example, a contract document generated from Word is identified as a machine-generated PDF because it contains an editable text layer; a scanned bill image taken with a mobile phone is identified as a traditional scanned PDF because it only contains image data; and an ID card scanned and then processed with OCR to add a text layer is identified as an OCR-enabled scanned PDF.

[0080] Step 2: Mixed Layout Processing

[0081] For each page of different types of PDF files, the tool combines layout attributes to achieve precise correction of mixed layouts, overcoming the limitation of existing tools that "only judge the first page";

[0082] Basis: The judgment criterion is based on the detection results of the Rotate attribute (which records the page rotation angle) of the PDF page;

[0083] Specific operations:

[0084] Attribute reading and supplementation: Read the Rotate attribute of each page, and combine it with the rotation direction information in the PDF file metadata (such as the "RotateMetadata" field) to avoid misjudgment based on a single attribute;

[0085] Precise correction: Adjust the layout angle of each page individually. For example, if the Rotate property is 90° and the text direction is detected as vertical, rotate it -90° to horizontal; if the Rotate property is 180° and the text direction is reversed, rotate it 180° to normal.

[0086] Step 3: Handling Seal Interference

[0087] For PDF pages containing stamps, the "pixel-level color separation - lightweight algorithm repair" technology can efficiently remove stamps while protecting text edges and table details to the greatest extent, balancing processing effect and efficiency.

[0088] Color feature separation: such as Figure 2 and Figure 3 As shown:

[0089] Stamp Removal (Based on Subject Color Detection, Applicable to Stamps of Any Color) Core Idea: Based on the assumption of black text on a white background:

[0090] Document text: Black / Gray → All three channel values ​​are low (R / G / B are not high)

[0091] Paper background: White / light color → All three channel values ​​are relatively high, but relatively balanced (R≈G≈B)

[0092] Seals usually have obvious color deviations, for example:

[0093] Red stamp: R channel is much higher than G and B.

[0094] Blue badge: Channel B is much higher than channels R and G.

[0095] Green badge: G channel is much higher than R and B.

[0096] The algorithm determines whether a pixel is the color of the stamp by combining two types of conditions:

[0097] 1. Absolute brightness threshold: A channel value > channel_threshold or > channel_high

[0098] 2. Color ratio threshold: The ratio of this channel to other channels > ratio_threshold

[0099] If the channel value is high enough and the color bias is obvious, it is identified as a stamp pixel and then whitened.

[0100] Detects three main seal colors: red, blue, and green;

[0101] By using dual detection of channel absolute value and color ratio, it avoids the complex processing of conventional color space conversion, morphological optimization (noise processing), contour detection and analysis, edge protection, and image restoration. While improving performance by 10 times or more, the accuracy of extracting text covered by stamps is also close to 100%, which is especially suitable for extracting PDFs with scanned tables and other images where the text and table lines are particularly thin.

[0102] Algorithm optimization and fixes:

[0103] A lightweight repair algorithm based on pixel replacement is adopted. For the separated stamp area, the stamp pixels are replaced by the surrounding background pixels (such as white background pixels), avoiding text loss and table breakage caused by traditional noise processing, contour detection and inpaint repair algorithms.

[0104] Performance metrics: The loss rate of text clarity after repair is less than 5%, the stamp residue rate is less than 1%, the processing speed is 10 times faster than traditional professional color space conversion methods, and it is suitable for batch processing needs.

[0105] The algorithm for removing the stamp is implemented using a pixel-level joint thresholding model, and specifically includes the following sub-steps:

[0106] I. Symbol Definition: Let the input color image be I. For any pixel position p, its BGR three-channel value is Define the mean of non-target channels:

[0107]

[0108]

[0109]

[0110] Define channel advantage ratio:

[0111] II. Threshold Parameter Settings: Set the channel absolute intensity threshold T, channel dominance ratio threshold K, and channel highlight threshold H;

[0112] III. Seal Pixel Discrimination: The red, blue, and green seal pixels are distinguished separately. The discrimination formula for the red seal pixels is:

[0113]

[0114] The discriminant for the blue stamp pixels is:

[0115]

[0116] The discriminant for the green stamp pixels is:

[0117]

[0118] The seal mask is obtained by taking the union of the three discrimination results.

[0119] IV. Mask Replacement and Background Fill: Perform background filling on pixels identified as stamps, and output the image:

[0120]

[0121] Furthermore, the channel absolute intensity threshold T corresponds to channel_threshold, the channel highlight threshold H corresponds to channel_high, and the channel advantage ratio threshold K corresponds to ratio_threshold;

[0122] In the process of determining the stamp pixels, when the mean value of the non-target channels is 0, the channel dominance ratio is set to 0 or a conditional division is performed according to the engineering implementation.

[0123] The algorithm is a pixel-by-pixel constant-time operation with an overall complexity of O(N), where N is the number of pixels in the image.

[0124] Explanation of the correspondence with actual scenarios / effects

[0125] Protecting the main text: The pixels of the main text are usually black / gray, and the overall values ​​of the three channels are low. In common cases, R, G, and B are difficult to exceed the absolute threshold T, thus failing to satisfy [C(p)>T], reducing the probability of mistaking the text for a seal.

[0126] Effective for multi-color stamps: When the stamp color is not fixed, it can still be classified into the case of "one channel is significantly larger than the other two channels", and the relative dominance relationship can be captured by ρ; the union of the three discriminant sets of red / blue / green covers the main colored stamps.

[0127] The significance of dual triggering: [ρC(p)>K] is more sensitive to "light stamps / semi-transparent stamps"; [C(p)>H] is more robust to "high saturation / high concentration stamps". Both can be used to take into account the differences in ink concentration and scanning exposure.

[0128] Value to the PDF preprocessing chain: Stamp removal is performed before / between page layout correction, classification and segmentation, and OCR / field extraction, which can reduce the probability of "mis-segmentation, misalignment, and missing characters caused by stamp coverage", thereby improving the stability of structured extraction; and the model is a pixel-by-pixel constant-time operation with an overall complexity of O(N) (N is the number of pixels), which is suitable for batch PDF page processing.

[0129] Step 4: Classify and process large files, such as... Figures 6 to 8 As shown:

[0130] For files of different sizes, especially large PDF files that meet the criteria of "file size ≥ 100MB or number of pages ≥ 100", the system achieves automated classification through "title extraction - intelligent classification - sub-file generation", reducing manual intervention and improving processing efficiency.

[0131] Title extraction: Differentiated extraction methods are used based on PDF type.

[0132] Machine-generated PDFs: Directly read text from the text layer located in the header area of ​​the page (usually the 1 / 3 area of ​​the page), with a font size larger than the body text size (e.g., if the body text is 12pt, then the title is ≥16pt), with a bold font style, no punctuation, and a character length ≤20 characters, and use it as the title information;

[0133] Intelligent classification:

[0134] Users can pre-input a set of file types to be categorized (e.g., when processing bank documents, input {"loan contract", "repayment record", "ID card"}).

[0135] Calculate the matching degree between the extracted title information and each type in the type set (e.g., the matching degree between "personal loan contract" and "loan contract" is ≥80%), and the type with the highest matching degree is the category to which the page belongs;

[0136] Sub-file generation and verification:

[0137] Record the original PDF file page range corresponding to each category (e.g., "Loan Contracts correspond to P1-P30" and "Repayment Records correspond to P31-P150"), generate independent sub-files, and keep the sub-file format consistent with the original PDF file format (e.g., PDF / A-1a, PDF / X-1a);

[0138] Efficiency and accuracy: Classification accuracy ≥90%, processing time reduced from the traditional full-text OCR-based classification method (approximately 10 minutes / 100 pages) to less than 1 second / 100 pages.

[0139] Step 5: Output preprocessing results

[0140] The processing results from steps two through four are integrated, and a preprocessed file is output after quality inspection to ensure the quality of use in downstream applications.

[0141] In summary, the preprocessing method for this PDF file is as follows:

[0142] Improved processing accuracy: Mixed layout correction accuracy ≥98%, stamp residue rate ≤1%, text clarity loss rate ≤5%, providing high-quality images for subsequent OCR recognition, improving recognition accuracy by 30%-50%, and avoiding business data errors caused by layout or stamp issues;

[0143] Improved processing efficiency: The processing time for large file classification has been reduced from more than 10 minutes to 1 second. The processing speed for mixed typesetting and stamp removal is 10 times faster than existing professional methods, meeting the efficiency needs of enterprises for batch processing (such as banks processing thousands of pages of documents every day).

[0144] Reduced processing costs: Based on an open-source technology framework, there are no additional hardware costs. The processing cost per thousand pages is less than 1 / 9 of that of commercial tools (such as Tencent Cloud OCR ¥180 / thousand pages), while avoiding repeated processing costs caused by recognition errors.

[0145] Expanding applicable scenarios: Applicable to three PDF types: machine-generated, traditional scanning, and scanning with OCR, covering mainstream documents such as enterprise contracts, invoices, and identity documents, providing reliable data preprocessing support for AI knowledge base construction and RAG technology implementation, and helping enterprises with digital transformation.

[0146] Core code for handling seals

[0147] def remove_stamps_main_colors(self, cv_image: np.ndarray) ->np.ndarray:

[0148] """

[0149] Remove stamps (based on main color detection, applicable to stamps of any color)

[0150] Core idea: based on the assumption of black text on a white background

[0151] - Detects three main stamp colors: red, blue, and green.

[0152] - Dual detection via channel absolute value + color ratio

[0153] - The text pixel (black / gray) values ​​in all channels are low, so it will not be misprocessed.

[0154] Detection logic:

[0155] A channel value > channel_threshold and (color ratio > ratio_threshold or channel value > channel_high)

[0156] Args:

[0157] cv_image: An image in OpenCV format (BGR format)

[0158] Returns:

[0159] Image after removing the stamp

[0160] """

[0161] # Get the parameters corresponding to the quality type

[0162] params = self.stamp_detection_params[self.quality_type]

[0163] channel_threshold = params['channel_threshold']

[0164] ratio_threshold = params['ratio_threshold']

[0165] channel_high = params['channel_high']

[0166] # Separate BGR channels

[0167] b, g, r = cv2.split(cv_image)

[0168] # Convert to float32 for accurate calculation

[0169] b_f = b.astype(np.float32)

[0170] g_f = g.astype(np.float32)

[0171] r_f = r.astype(np.float32)

[0172] #Red Detection

[0173] avg_gb = (g_f + b_f) / 2

[0174] red_ratio = np.divide(r_f, avg_gb, out=np.zeros_like(r_f), where=avg_gb > 0)

[0175] is_red = (r > channel_threshold) & ((red_ratio > ratio_threshold) |(r > channel_high))

[0176] # Blue Detection

[0177] avg_rg = (r_f + g_f) / 2

[0178] blue_ratio = np.divide(b_f, avg_rg, out=np.zeros_like(b_f), where=avg_rg > 0)

[0179] is_blue = (b > channel_threshold) & ((blue_ratio > ratio_threshold) |(b > channel_high))

[0180] # Green Testing

[0181] avg_rb = (r_f + b_f) / 2

[0182] green_ratio = np.divide(g_f, avg_rb, out=np.zeros_like(g_f), where=avg_rb > 0)

[0183] is_green = (g > channel_threshold) & ((green_ratio > ratio_threshold)| (g > channel_high))

[0184] # Stamp requirements: Must meet any of the following conditions: red / blue / green

[0185] is_stamp = is_red | is_blue | is_green

[0186] # Set the detected stamp area to white

[0187] cv_image[is_stamp] = 255

[0188] return cv_image

[0189] Determine if it is a scanned document

[0190] ef _is_scanned_page(self, page) -> bool:

[0191] """Determine if the page is a scanned document (full-page image)""

[0192] text_content = ''

[0193] image_elements = sum(1 for element in page if isinstance(element, LTFigure))

[0194] total_elements = sum(1 for _ in page)

[0195] if total_elements == 0:

[0196] return False

[0197] # If image elements occupy more than 80% of the page and the text content is relatively small, it is considered a scanned page.

[0198] is_mostly_image = (image_elements / total_elements) > 0.8

[0199] if is_mostly_image:

[0200] text_content = ''.join(element.get_text() for element in page

[0201] if hasattr(element, 'get_text'))

[0202] return len(text_content.strip()) < 50

[0203] return False

[0204] Core code for judging layout angle

[0205] def _detect_rotation(self, image: Image.Image) -> int:

[0206] """Detecting image rotation angle"""

[0207] try:

[0208] osd_info = image_to_osd(image, config='--psm 0 -l osd')

[0209] if 'Rotate: ' in osd_info:

[0210] content_rotation = int(osd_info.split('Rotate: ')[1].split('\n')[0])

[0211] if content_rotation != 0:

[0212] rotation = content_rotation % 360

[0213] return rotation if rotation in [0, 90, 180, 270] else 0

[0214] except Exception as e:

[0215] print(f"OCR direction detection failed: {str(e)}")

[0216] width, height = image.size

[0217] if width > height * 1.2:

[0218] return 90

[0219] return 0

[0220] Intelligent identification of mixed types (extracting types for easy segmentation and classification) core code

[0221] def extract_titles(pdf_path):

[0222] """

[0223] Extract the title of each page from the PDF and categorize it.

[0224] parameter:

[0225] pdf_path: PDF file path

[0226] return:

[0227] titles: A dictionary containing the title of each page {page number: title}

[0228] categories: A list of page numbers categorized by category {category: [page number list]}

[0229] """

[0230] titles = {}

[0231] categories = defaultdict(list)

[0232] current_category = None

[0233] # Iterate through every page of the PDF

[0234] for page_num, page in enumerate(extract_pages(pdf_path), 1):

[0235] page_title = None

[0236] max_font_size = 0

[0237] first_text_block = None

[0238] text_blocks = []

[0239] page_height = page.bbox[3]

[0240] page_width = page.bbox[2] # Get page width

[0241] # Traverse all text elements on the page

[0242] for element in getattr(page, "_objs", []):

[0243] if isinstance(element, LTTextContainer):

[0244] # Get the text content of this text block

[0245] text = element.get_text().strip()

[0246] if not text:

[0247] continue

[0248] # Record the first text block

[0249] if first_text_block is None:

[0250] first_text_block = text

[0251] # Get the maximum font size and position information in this text block

[0252] current_font_size = max((char.size for text_line in element

[0253] for char in text_line

[0254] if isinstance(char, LTChar)), default=0)

[0255] # Calculate the relative position of a text block on the page (between 0 and 1).

[0256] relative_position_y = element.y1 / page_height

[0257] # Calculate the horizontal center position of the text block

[0258] text_center_x = (element.x0 + element.x1) / 2

[0259] relative_position_x = text_center_x / page_width

[0260] # Determine if the text is centered (allowing for some deviation)

[0261] is_centered = TITLE_CONFIG['CENTER_RANGE'][0] <= relative_position_x<= TITLE_CONFIG['CENTER_RANGE'][1]

[0262] # Clean up the text, remove special characters and extra spaces

[0263] cleaned_text = re.sub(r'[^\u4e00-\u9fa5a-zA-Z0-9]', ' ', text).strip()

[0264] cleaned_text = re.sub(r'\s+', '', cleaned_text)

[0265] # Characteristics for determining whether it is a title

[0266] is_title = False

[0267] if cleaned_text is in settings.MATERIAL_PDF_SPLIT_TYPE_LIST: # First check if it is in the predefined heading list.

[0268] if is_centered and (page_title is None): # Ensures the page title is centered and is appearing for the first time.

[0269] is_title = True

[0270] max_font_size = current_font_size

[0271] page_title = cleaned_text

[0272] elif current_font_size > max_font_size and is_centered:

[0273] # 1. Length characteristic: Headlines are usually short

[0274] if len(cleaned_text) < 15:

[0275] # 2. Content Characteristics: Excluding Specific Patterns

[0276] if not any([

[0277] re.match(r'^[\d.]+$', cleaned_text), # Excludes pure numbers

[0278] re.match(r'^[\[\]\{\}\"\'\.。,,]+$', cleaned_text), # Exclude pure symbols

[0279] cleaned_text.startswith('{'), # Exclude the beginning of the JSON string.

[0280] cleaned_text.endswith('。'), # Exclude sentences ending with a period.

[0281] re.search(r'[。!?,;].*[。!?,;]', cleaned_text), # Exclude long sentences containing multiple punctuation marks

[0282] `len(cleaned_text.split()) > 10`, # Exclude sentences with excessive word count.

[0283] ':' in cleaned_text and len(cleaned_text.split(':')[1]) > 30 # Exclude sentences with excessively long content after the colon ]):

[0285] # 3. Location characteristics: Titles are usually at the top of the page.

[0286] if relative_position_y > 0.8 and page_title is None:

[0287] print(f"Page {page_num}, suspected title found: {cleaned_text}")

[0288] # is_title = True

[0289] # max_font_size = current_font_size

[0290] # page_title = text

[0291] text_blocks.append((text, current_font_size, relative_position_y, is_centered))

[0292] # Title cleanup and validation

[0293] if page_title:

[0294] # Remove extra whitespace characters and newline characters

[0295] page_title = re.sub(r'\s+', ' ', page_title)

[0296] # Limit header length

[0297] page_title = page_title[:100] + ('...' if len(page_title) > 100 else'')

[0298] # Verify the title's validity

[0299] if len(page_title.strip()) < 3 or page_title.count('{') > 0:

[0300] page_title = 'None'

[0301] else:

[0302] page_title = 'None'

[0303] titles[page_num] = page_title

[0304] # Category Classification

[0305] if page_title != 'None':

[0306] # Update current category

[0307] current_category = page_title

[0308] categories[current_category].append(page_num)

[0309] elif current_category: # Untitled pages are moved to the previous category

[0310] categories[current_category].append(page_num)

[0311] return titles, categories

[0312] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A PDF file preprocessing method, characterized in that, Specifically, the following steps are included: Step 1: PDF Type Determination: Obtain the PDF file to be processed and determine the type of the PDF file; Step 2, Mixed Layout Processing: For each page of the PDF file, the layout direction is determined and corrected based on the relevant layout attributes. Step 3: Seal Interference Handling: For PDF file pages containing seals, the seals are removed and the text content is protected through color feature separation and algorithm optimization. Step 4: Large file classification and processing: For files of different sizes or number of pages, especially large PDF files, extract page feature information, classify the files according to the feature information, and generate sub-files of the corresponding types; Step 5: Output preprocessing results: Integrate the processing results from Steps 2 to 4, and output the preprocessed PDF file.

2. The PDF file preprocessing method according to claim 1, characterized in that, In step one, the types of PDF files include machine-generated PDFs, traditional scanned PDFs, and scanned PDFs with OCR; the type is determined by parsing the PDF file metadata, judging the image ratio, and the amount of text.

3. The PDF file preprocessing method according to claim 2, characterized in that, In step two, the layout-related attributes include the Rotate attribute of the PDF page; during correction, the Rotate attribute of each page is determined separately, and the page is adjusted to the correct layout angle.

4. The PDF file preprocessing method according to claim 1, characterized in that, In step three, the color feature separation includes the core idea of ​​removing the stamp (based on the main color detection, applicable to stamps of any color).

5. A PDF file preprocessing method according to claim 4, characterized in that, In step three, the seal is removed, based on the assumption of black text on a white background: Document text: Black / Gray → All three channel values ​​are low (R / G / B are not high) Paper background: White / light color → All three channel values ​​are relatively high, but relatively balanced (R≈G≈B) Seals usually have obvious color deviations, for example: Red stamp: R channel is much higher than G and B. Blue badge: Channel B is much higher than channels R and G. Green badge: G channel is much higher than R and B. The algorithm determines whether a pixel is the color of the stamp by combining two types of conditions:

1. Absolute brightness threshold: A channel value > channel_threshold or > channel_high 2. Color ratio threshold: The ratio of this channel to other channels > ratio_threshold If the channel value is high enough and the color bias is obvious, it is identified as a stamp pixel and then erased. By using dual detection of absolute channel value and color ratio, it avoids the complex algorithms and processing of conventional color space conversion, morphological optimization (noise processing), contour detection and analysis, edge protection, and image restoration. While improving performance by 10 times or more, the accuracy of extracting text covered by stamps is close to 100%, which is particularly suitable for extracting text from scanned documents such as tables and PDFs with very thin table lines.

6. The PDF file preprocessing method according to claim 1, characterized in that, In step four, PDFs that meet different size or page number conditions are processed (generally, PDFs with a file size ≥ 100MB or a page count ≥ 100 pages cannot be processed). For PDFs of different sizes, the page feature information includes the title information on the page. The title information must meet at least one of the following characteristics: located in the header area of ​​the page, font size larger than the body text size of the page, bold font style, no punctuation marks, and character length ≤ 20 characters to extract type features.

7. A PDF file preprocessing method according to claim 6, characterized in that, In step four, when extracting title information, the title in the text layer of the machine-generated PDF is read directly (based on logical judgments such as position and text size); when classifying, a set of file types to be classified can be passed in in advance, and the classification is completed based on the matching degree between the title information and the set of types, with a classification accuracy of ≥90%.

8. A PDF file preprocessing method according to claim 7, characterized in that, In step four, when generating sub-files, the page range of the original PDF file corresponding to each sub-file is recorded, and the format of the sub-files is consistent with the format of the original PDF file.

9. A PDF file preprocessing method according to claim 5, characterized in that, The algorithm for removing the stamp in step 3 is implemented using a pixel-level joint thresholding model, and specifically includes the following sub-steps: I. Symbol Definition: Let the input color image be I. For any pixel position p, its BGR three-channel value is Define the mean of non-target channels: ； ； ； Define channel advantage ratio: ; II. Threshold Parameter Settings: Set the channel absolute intensity threshold T, channel dominance ratio threshold K, and channel highlight threshold H; III. Seal Pixel Discrimination: The red, blue, and green seal pixels are distinguished separately. The discrimination formula for the red seal pixels is: ； The discriminant for the blue stamp pixels is: ； The discriminant for the green stamp pixels is: ； The seal mask is obtained by taking the union of the three discrimination results. ; IV. Mask Replacement and Background Fill: Perform background filling on pixels identified as stamps, and output the image: 。 10. A PDF file preprocessing method according to claim 9, characterized in that, The absolute intensity threshold T of the channel corresponds to channel_threshold, the highlight threshold H of the channel corresponds to channel_high, and the advantage ratio threshold K of the channel corresponds to ratio_threshold. In the process of determining the stamp pixels, when the mean value of the non-target channels is 0, the channel dominance ratio is set to 0 or a conditional division is performed according to the engineering implementation. The algorithm is a pixel-by-pixel constant-time operation with an overall complexity of O(N), where N is the number of pixels in the image. A PDF file preprocessing method, characterized in that, Specifically, the following steps are included: Step 1: PDF Type Determination: Obtain the PDF file to be processed and determine the type of the PDF file; Step 2, Mixed Layout Processing: For each page of the PDF file, the layout direction is determined and corrected based on the relevant layout attributes. Step 3: Seal Interference Handling: For PDF file pages containing seals, the seals are removed and the text content is protected through color feature separation and algorithm optimization. Step 4: Large file classification and processing: For files of different sizes or number of pages, especially large PDF files, extract page feature information, classify the files according to the feature information, and generate sub-files of the corresponding types; Step 5: Output preprocessing results: Integrate the processing results from Steps 2 to 4, and output the preprocessed PDF file.

Images