Method for identifying PDF (Portable Document Format) file layout based on mask processing

Abstract

The invention discloses a PDF (Portable Document Format) file layout identification method based on mask processing, and aims to realize accurate identification of various layout elements such as titles, texts, page headers, page footers, tables and pictures in PDF files. According to the method, characters and coordinates in a PDF file are analyzed through MuPDF, different color masks are adopted to replace texts and punctuation marks respectively, original content of a non-text area is reserved, a page picture subjected to mask processing is generated, a training data set is constructed based on the page picture, training is conducted through a target detection model (such as YOLO), and a high-precision layout recognition model is obtained. According to the method, the accuracy and the automation level of PDF layout recognition are effectively improved, and the method has a wide application prospect.

Description

Technical Field

[0001] This invention relates to the fields of document processing and computer vision technology, specifically to a PDF file layout recognition method based on masking, which can be used to automatically identify layout elements such as titles, body text, headers, footers, tables, and images in PDF documents. Background Technology

[0002] PDF (Portable Document Format) files are widely used for document storage and distribution due to their cross-platform compatibility and format stability. However, the complex layout structure of PDF files (such as titles, body text, headers, footers, tables, and images) poses challenges to automated content extraction and analysis.

[0003] Traditional PDF layout recognition methods often rely on rule-based parsing or simple image processing techniques, making them ill-suited to diverse layout structures and resulting in low recognition accuracy. Furthermore, existing PDF layout recognition methods frequently require extensive manual annotation when handling complex layouts, leading to high costs and low efficiency.

[0004] In recent years, advancements in deep learning technology in object detection have provided new insights into PDF layout recognition. However, efficiently extracting layout features and constructing a training dataset remains a challenge.

[0005] Therefore, with the development and changes in technology and application scenarios, there is an urgent need in this field for innovative, efficient, accurate and highly automated PDF layout recognition methods to overcome the above-mentioned technical defects and other technical problems in existing technologies.

[0006] The information included in this background section of the specification of this invention, including any references cited herein and any descriptions or discussions thereof, is included for technical reference purposes only and is not intended to limit the scope of the invention. Summary of the Invention

[0007] The present invention is proposed in view of the foregoing and other further ideas. The present invention aims to solve the above-mentioned technical deficiencies and other technical problems.

[0008] The purpose of this invention is to provide an innovative, highly automated, highly accurate, and highly adaptable layout recognition method.

[0009] To address the shortcomings of existing technologies, this invention proposes a PDF file layout recognition method based on masking processing. By combining MuPDF parsing, masking processing technology, and object detection models, it achieves efficient recognition of various layout elements in PDF files.

[0010] To achieve the above objectives, the following technical means are employed according to one or more embodiments reflecting the main inventive concept of the present invention:

[0011] Step 1: Use the MuPDF tool to parse the PDF file and extract the text content and its corresponding text bounding box coordinates.

[0012] Step 2: For the text portion in the text area, use the first color block (such as black) to perform mask replacement, generating a text mask area.

[0013] Step 3: Use a second color block (such as red) to mask and replace the punctuation marks in the text area to generate a punctuation mask area.

[0014] Step 4: Preserve the original content of non-text areas (such as tables, images, etc.) in the PDF file to create a masked page image.

[0015] Step 5: Convert the PDF file to image format and map the parsed text coordinates to the image coordinate system; based on the masked page image, annotate the layout elements such as titles, body text, headers, footers, tables, and images to generate a training dataset.

[0016] Step 6: Use an object detection model (such as the YOLO model) to train the training dataset, optimize the model parameters, and obtain the layout recognition model.

[0017] Compared with the prior art, the present invention can provide the following advantages and beneficial technical effects:

[0018] High precision: Enhances the features of layout elements through masking, thereby improving the recognition accuracy of the target detection model.

[0019] High automation: Automated parsing and dataset generation reduce human intervention and improve efficiency.

[0020] Highly adaptable: It can handle diverse PDF layouts and is suitable for different types of documents.

[0021] Scalability: The method of this invention is highly versatile and can be extended to layout recognition tasks of other document formats.

[0022] More specifically, according to the present invention, a method for PDF file layout recognition based on masking is provided, comprising: a) parsing the PDF file using the MuPDF tool to extract the text content and its corresponding text bounding box coordinates in the PDF file; b) masking the text portion in the text region of the PDF file using a first color block to generate a text mask region; c) masking the punctuation marks in the text region using a second color block to generate a punctuation mask region; d) retaining the original content of the non-text regions in the PDF file to form a masked page image; e) converting the PDF file into an image format and proportionally scaling the parsed text bounding box coordinates for coordinate mapping to an image coordinate system; f) labeling the layout elements of the PDF file based on the masked page image to generate a training dataset; g) training the training dataset using an object detection model, optimizing the model parameters, and obtaining a layout recognition model.

[0023] According to one embodiment of the present invention, the first color is black (RGB: 0, 0, 0) and the second color is red (RGB: 255, 0, 0).

[0024] According to one embodiment of the present invention, the target detection model is a YOLO series model, preferably the YOLO v5 model.

[0025] According to one embodiment of the present invention, the non-text area includes a table area and an image area.

[0026] According to one embodiment of the present invention, the layout elements include at least one of the following in the PDF file: title, body text, header, footer, table, and image.

[0027] According to one embodiment of the present invention, the annotation process of the training dataset adopts a semi-automated annotation method.

[0028] According to an embodiment of the present invention, the hyperparameter settings for training the training dataset using the object detection model include: learning rate 0.01 and batch size 16.

[0029] According to one embodiment of the present invention, the result of parsing the PDF file is stored in JSON format.

[0030] According to one embodiment of the present invention, the resolution of the page image after masking is 300 DPI or higher.

[0031] According to one embodiment of the present invention, the output of the layout recognition model includes the bounding box coordinates and category labels of various layout elements.

[0032] According to one embodiment of the present invention, the non-text area includes a table area, an image area, a chart area, and a graphic area, which retain their original pixel content during the masking process.

[0033] According to one embodiment of the present invention, the YOLO object detection model is a YOLOv5 model, the input image size is 640×640 pixels, it is trained using the SGD optimizer, the momentum parameter is 0.937, and the weight decay coefficient is 0.0005.

[0034] According to one embodiment of the present invention, the annotation of the page elements adopts the PASCAL VOC format or COCO format, and the annotation categories include eight categories such as title, body text, header, footer, table, picture, list, and note.

[0035] According to one embodiment of the present invention, the loss function for model training includes classification loss, bounding box regression loss, and target confidence loss, with weights of α = 0.5, β = 0.05, and Y = 1.0, respectively.

[0036] According to one embodiment of the present invention, the non-maximum suppression (NMS) threshold for the model application phase is 0.5, and the confidence threshold is 0.3.

[0037] According to one embodiment of the present invention, the PDF file parsing result is stored in JSON format, including text content, bounding box coordinates, font information, font size, and color information.

[0038] According to one embodiment of the present invention, the page image after masking is in PNG format, with a resolution of 300 DPI and a color mode of RGB.

[0039] According to one embodiment of the present invention, the scaling factor of the coordinate mapping ranges from 0.8 to 1.2, with a default value of 1.0.

[0040] According to one embodiment of the present invention, the output of the layout recognition model includes the bounding box coordinates, category labels, and confidence scores of various layout elements, with the confidence scores ranging from 0 to 1.

[0041] According to one embodiment of the present invention, the method is applicable to various types of PDF documents such as academic papers, business documents, technical manuals, legal documents, and medical records.

[0042] According to one embodiment of the present invention, the method supports the processing of single-page and multi-page PDF documents, with a processing speed of less than 1 second per page on average, preferably less than 0.8 seconds.

[0043] According to one embodiment of the present invention, the punctuation marks include commonly used punctuation marks such as period, comma, semicolon, colon, quotation mark, parentheses, question mark, and exclamation mark.

[0044] According to one embodiment of the present invention, the annotation accuracy of the training dataset is required to reach more than 95%, and a cross-validation mechanism is used to ensure the annotation quality.

[0045] According to an embodiment of the present invention, the average F1 score of the layout recognition model reaches over 92%, which is significantly better than traditional rule-based PDF layout recognition methods.

[0046] Further embodiments of the present invention can achieve other advantageous technical effects not listed hereafter, which may be partially described below and can be expected and understood by those skilled in the art after reading the present invention.

[0047] The summary portion of this invention is intended to introduce, in a simplified form, the concepts and options that will be further described in the "Detailed Description" section below to aid the reader in understanding the invention. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. All of the foregoing features are to be understood as exemplary only. A more complete display of the features, details, utility, and advantages of the invention will be provided in the following written description of various embodiments of the invention, illustrated in the accompanying drawings, and defined in the appended claims. Therefore, the many limiting interpretations of the summary cannot be understood without further reading of the entire specification, claims, and drawings. Attached Figure Description

[0048] The above-described features and advantages of these embodiments, as well as other features and advantages, and the ways in which they are implemented, will become more apparent from the following description in conjunction with the accompanying drawings, and the invention will be better understood.

[0049] Figure 1 The diagram schematically illustrates the overall system architecture of PDF file layout recognition based on masking processing according to an embodiment of the present invention, including data flow and processing modules.

[0050] Figure 2 The diagram illustrates the specific process of masking and the color coding scheme.

[0051] Figure 3 This schematically illustrates the training process and key steps of the YOLO model.

[0052] Figure 4 The schematic illustration shows the operation flow and output results of layout recognition in an example of the present invention. Detailed Implementation

[0053] In the following description of specific embodiments, details of one or more embodiments of the present invention will be set forth. Other features, objects, and advantages of the invention will become apparent from these descriptions and the claims.

[0054] It should be understood that the described embodiments are not limited in application to the details set forth in the following description. The illustrated embodiments may be other embodiments and can be implemented or performed in various ways. The examples are provided by way of explanation rather than limitation of the disclosed embodiments. Indeed, it will be apparent to those skilled in the art that various modifications and variations can be made to the embodiments of the invention without departing from the scope or spirit of the invention disclosure. Therefore, this disclosure covers such modifications and variations that fall within the scope of the appended claims and their equivalents.

[0055] Similarly, it should be understood that the phrases and terms used in this document are for descriptive purposes and should not be considered restrictive. For example, the use of “including,” “contains,” or “has,” and their variations, is intended to include, in an open-ended sense, the items listed thereafter, their equivalents, and any additional items.

[0056] The following will refer to specific embodiments of the present invention and accompanying drawings. Figure 1-4 The PDF file layout recognition method of the present invention will be explained and described in more detail.

[0057] Step 1: PDF file parsing

[0058] Use the MuPDF library (version 1.18.0 or later) to parse PDF files, extract the text content, font information, font size, color information and bounding box coordinates of each page, and store them as structured data (such as JSON format).

[0059] Output format: Structured JSON data, containing the following fields:

[0060] {"page_nm": 1, "text_elements": [{"text": "Example text", "bbox": [x1, y1, x2, y2], "font": "Arial", "size": 12, "color": "#000000"}]}

[0061] Step 2: Masking

[0062] 2.1 Text Mask: The text in the text area is covered with a black (RGB: 0, 0, 0) color block to generate a text mask area.

[0063] 2.2 Punctuation Mask: The punctuation mark area is covered with a red (RGB: 255, 0, 0) color block to generate a punctuation mask area.

[0064] 2.3 Non-text areas: Preserve the original pixel content of non-text areas (such as tables, pictures, charts, etc.) in the PDF file to form a complete masked page image.

[0065] 2.4 Mask parameter settings: The size of the mask color block is exactly the same as the original text bounding box to ensure complete coverage of the original text content.

[0066] Step 3: PDF to Image Conversion and Coordinate Mapping

[0067] Use the Poppler toolset (version 0.68.0 or later) or MuPDF's pdfdraw tool to convert the PDF file to a high-resolution image, such as a PNG format, at 300 DPI or higher, ensuring the clarity of text and layout elements; then map the parsed text coordinates from the PDF coordinate system to the image coordinate system using the following formula:

[0068] x_image=x_pdf×(DPI / 72)×scale_factor

[0069] y_image=y_Pdf×(DPI / 72)×scale_factor

[0070] Here, scale_factor is the scaling factor, which defaults to 1.0.

[0071] Step 4: Dataset Labeling

[0072] Based on the masked page images, layout elements such as titles, body text, headers, footers, tables, and images are labeled using semi-automatic or manual methods to generate the training dataset required for object detection.

[0073] Dataset labeling categories: including at least 8 categories such as title, body text, header, footer, table, image, list, and annotation.

[0074] Annotation format: Use PASCAL VOC or COCO format. Each annotation contains the following information:

[0075] {"bbox":[x,y,width,height],"category":"title","confidence":0.95}

[0076] The annotation accuracy rate is required to reach over 95%, and a cross-validation mechanism is used to ensure annotation quality.

[0077] Step 5: Model Training

[0078] Using a YOLO model, such as YOLO v5, and inputting a training dataset, hyperparameters are set (e.g., learning rate 0.01, batch size 16). The model is then optimized through multiple iterations to obtain the layout recognition model. The specific configuration is as follows:

[0079] Input dimensions 640×640 pixels Model input image size Batch size 16 Batch size during training Learning rate 0.01 Initial learning rate Number of training rounds 300 Total number of training rounds Optimizer SGD Stochastic gradient descent optimizer momentum 0.937 SGD optimizer momentum parameters Weight decay 0.0005 L2 regularization coefficient

[0080] Step 6: Model Application and Inference

[0081] Inference process: Process the PDF file to be identified according to steps 1-3 to generate a mask image, and then input it into the trained YOLO model for inference.

[0082] Output: The model outputs the bounding box coordinates, category labels, and confidence scores for various layout elements.

[0083] Post-processing: Non-maximum suppression (NMS) algorithm is used to remove duplicate detection boxes, and the NMS threshold is set to 0.5.

[0084] Confidence threshold: Set the confidence threshold to 0.3, and only retain detection results with a confidence level higher than this threshold.

[0085] Test Results

[0086] Experiments were conducted on a PDF dataset containing various page layouts. Our method performed exceptionally well in recognizing titles, body text, headers, footers, tables, and images, achieving an average F1 score of over 92%, which is superior to traditional rule-based methods and simple image processing techniques.

[0087] The technical advantages of this invention are listed below:

[0088] High-precision recognition: Enhanced feature of layout elements through masking improves the recognition accuracy of the target detection model, with an average F1 score of over 92%.

[0089] High degree of automation: Automated parsing and dataset generation reduce human intervention and improve efficiency by more than 80%.

[0090] Highly adaptable: It can handle diverse PDF layouts and is suitable for different types of documents, including academic papers, technical documents, and business reports.

[0091] Scalability: The method is highly versatile and can be extended to layout recognition tasks for other document formats.

[0092] Real-time processing: The processing time for a single-page PDF document is less than 1 second, meeting the requirements for real-time processing.

[0093] The implementation process of this invention will be explained in detail below through more application examples.

[0094] Application Example 1: Academic Paper Layout Recognition

[0095] The implementation environment configuration is as follows:

[0096] Components Version / Model Configuration parameters operating system Ubuntu 20.04LTS 64-bit system Python 3.8.5 Programming Language Environment MuPDF 1.18.0 PDF parsing library PyTorch 1.9.0 Deep learning framework YOLOv5 5.0 Object detection model CUDA 11.1 GPU acceleration library

[0097] The specific steps are as follows:

[0098] Step 1: Collect 1000 PDF files of academic papers from different journals as a training dataset, covering multiple disciplines such as computer science, physics, and biology.

[0099] Step 2: Use MuPDF to parse the PDF file, extract the text content and bounding box coordinates. Example code is as follows:

[0100]

[0101]

[0102] Step 3: Perform masking to generate training images. Key parameter settings are as follows:

[0103] Parameter name Parameter value Range of values Image resolution 300 DPI 150-600 DPI Text mask color RGB(0, 0, 0) Black series Punctuation mask color RGB(255,0,0) Red series Image format PNG PNG, JPG, TIFF

[0104] Step 4: Model training and optimization, monitoring key parameters during training: Loss function: Loss = α × L_cls + β × L_box + γ × L_obj

[0105] in:

[0106] L_cls: Classification loss (cross-entropy loss)

[0107] L_box: Bounding box regression loss (CIoU loss)

[0108] L_obj: Target confidence loss

[0109] α = 0.5, β = 0.05, γ = 1.0 (empirical parameters)

[0110] Application Example 2: Business Document Layout Recognition

[0111] Application scenarios: Automated processing of business documents such as corporate financial statements, business plans, and market research reports.

[0112] Training data: Collect 500 different types of business documents, including annual reports, quarterly reports, business plans, etc.

[0113] Annotation category expansion: Add special layout element categories such as "Charts", "Data Tables", "Page Numbers", and "Company Logos".

[0114] Model Adjustment: To address the characteristics of commercial documents, the anchor box size of the YOLO model was adjusted to enhance its ability to detect small targets (such as page numbers and annotations).

[0115] Post-processing optimization: Added layout logic relationship analysis to identify the document's hierarchical structure.

[0116] Application Example 3: Technical Manual Layout Recognition

[0117] Application scenarios: Digital processing of technical documents such as industrial equipment technical manuals, product manuals, and maintenance guides.

[0118] Multilingual support: Supports layout recognition for documents with mixed Chinese and English.

[0119] Special element identification: Focus on identifying special layout elements such as warning boxes, notices, and step lists.

[0120] Structured output: Outputs structured document hierarchical information, facilitating integration with subsequent document management systems.

[0121] Quality assessment: Provides confidence assessment for identification, and allows for manual verification of identification results with low confidence.

[0122] Detailed description of key technical parameters

[0123] The optimal parameter combination was determined through extensive experiments:

[0124]

[0125]

[0126] The foregoing description of embodiments of the invention has been provided for illustrative purposes. This foregoing description is not intended to be exhaustive, nor is it intended to limit the invention to the precise features and / or forms disclosed. Clearly, many modifications and variations can be made in light of the teachings above, all of which fall within the scope of this invention. The scope of the invention is defined only by the appended claims.

Claims

1. A method for PDF file layout recognition based on mask processing, characterized in that, The method comprises: a) parsing a PDF file using a MuPDF tool to extract the text content and its corresponding text bounding box coordinates in the PDF file; b) using a first color block to mask replace the text part in the text area in the PDF file to generate a text mask area; c) using a second color block to mask replace the punctuation part in the text area to generate a punctuation mask area; d) retaining the original content of the non-text area in the PDF file to form a page picture after mask processing; e) converting the PDF file into a picture format and scaling the parsed text bounding box coordinates proportionally for coordinate mapping into a picture coordinate system; f) annotating the layout elements of the PDF file based on the page picture after mask processing to generate a training data set; and g) training the training data set using a target detection model, optimizing the model parameters to obtain a layout recognition model.

2. The method of claim 1, wherein, The first color is black (RGB: 0, 0, 0), and the second color is red (RGB: 255, 0, 0).

3. The method of claim 1, wherein, The target detection model is a YOLO series model, preferably a YOLO v5 model.

4. The method of claim 1, wherein, The non-text area includes a table area and a picture area.

5. The method of claim 1, wherein, The layout elements include at least one of the title, the main text, the header, the footer, the table and the picture in the PDF file.

6. The method of claim 1, wherein, The annotation process of the training data set adopts a semi-automatic annotation method.

7. The method of claim 1, wherein, The hyperparameter settings for training the training data set using the target detection model include a learning rate of 0.01 and a batch size of 16.

8. The method of claim 1, wherein, The results of parsing the PDF file are stored in JSON format.

9. The method of claim 1, wherein, The resolution of the page picture after mask processing is 300DPI or higher.

10. The method of claim 1, wherein, The output of the layout recognition model includes the bounding box coordinates and class labels of each type of layout element.

Images