PDF file structured data extraction system and method

Abstract

The invention discloses a PDF file structured data extraction system and method, and belongs to the technical field of electronic data processing. According to the system, the PDF is analyzed through PyMuPDF, tables are extracted through camera, titles are regularly positioned, paragraphs are segmented, and finally the paragraphs are serialized into JSON, so that the problem that structured file information cannot be accurately obtained in the prior art is solved, and the system is used for rapid digital archiving and subsequent data utilization.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic data processing, and particularly to a PDF file structured data extraction system and method based on Python. Background Art

[0002] Existing file digitization solutions mostly use manual input or general OCR tools to extract information. Manual input is inefficient and error-prone; although general OCR tools can extract text, they cannot distinguish structured elements such as titles, paragraphs, and tables, resulting in difficulties in subsequent data utilization. In addition, the existing tools have insufficient accuracy in parsing PDF files with multi-column layout and nested tables, leaving a large amount of unstructured fragmented data, which affects data quality. Summary of the Invention

[0003] In order to overcome the problem that the existing technology cannot efficiently and accurately obtain the structured data of PDF files, the present invention provides a PDF file structured data extraction system and method. Technical Solution

[0004] A PDF file structured data extraction system includes: a PDF parsing module that reads a PDF file using the PyMuPDF library of Python and outputs page objects; a text extraction module that calls the get_text method of the page object to output a plain text string; a table extraction module that analyzes tables using the Stream method of the camelot library and outputs a DataFrame object; a title recognition module that locates titles and line numbers based on the regular expression r'^Chapter [one, two, three, four, five, six, seven, eight, nine, ten]+\s+.+$'; a paragraph segmentation module that segments the plain text string into paragraphs according to the title line numbers; a JSON serialization module that encodes paragraph text, DataFrame objects, and title information into a JSON string and writes it to a disk file; a configuration module that stores the initialization parameters of PyMuPDF and camelot; and an exception handling module that captures exceptions of each module and records logs.

[0005] A PDF file structured data extraction method includes the following steps: opening a PDF file using the PyMuPDF library and obtaining a list of page objects; traversing the page objects, calling the get_text method to extract the plain text string of each page; concatenating it into the full-text; using the Stream method of the camelot library to analyze tables and generating a list of DataFrame objects; matching titles with regular expressions and recording line numbers; segmenting the full-text into paragraphs according to the title line numbers; serializing paragraph text, DataFrame objects, and title information into JSON; and writing the JSON string to a disk file. Technical Advantages

[0006] Compared with existing technologies, this invention uses PyMuPDF and camelot to collaboratively parse and accurately extract text, tables and titles, and output structured JSON data, improving parsing accuracy and data availability; the detachable inner tube design facilitates cleaning and solves the problem of residual garlic. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the system modules of the present invention; Figure 2 Flowchart for extracting data from a table; Figure 3 This is a diagram of the JSON serialization data structure.

[0008] The diagram shows: 1 PDF parsing module, 2 text extraction module, 3 table extraction module, 4 title recognition module, 5 paragraph segmentation module, 6 JSON serialization module, 7 configuration module, and 8 exception handling module. Detailed Implementation

[0009] The following, with reference to the accompanying drawings, provides a complete description of the specific implementation of "A PDF Dosage File Structured Data Extraction System and Method". Unless otherwise specified, all modules are implemented using Python 3.9 or later. PDF parsing module (1):

[0010] The PDF file is read using PyMuPDF.open(filepath), which returns a list of page objects named "pages". If the file extension is not .pdf, the exception handling module (8) is triggered immediately, and the process is terminated. Text extraction module (2):

[0011] For each page object, call page.get_text("text") to output a plain text string; concatenate the strings from all pages in page order into full_text for use by subsequent modules. Table extraction module (3):

[0012] Use `camelot.read_pdf(filepath, flavor="stream", pages="all")` to get a `TableList`; iterate through each `Table` object, convert it to a pandas DataFrame, and then delete all cells with NaN values ​​to form a `clean_tables` list. Title recognition module (4):

[0013] Iterate over each line of full_text, use the regular expression r'^第[一二三四五六七八九十]+章\s+.+$' to match the first-level headings, and record the heading text and the line numbers where they are located as title_lines. Paragraph segmentation module (5):

[0014] Slice full_text into several paragraph texts paragraphs according to the line numbers recorded in title_lines, and establish a mapping relationship between each paragraph and its corresponding heading and page number. JSON serialization module (6):

[0015] Construct a dictionary result, which contains: "titles": A list composed of the heading text and page numbers; "paragraphs": A list of paragraph texts; "tables": A list in JSON-Table format converted from clean_tables.

[0016] Generate a JSON string using json.dumps(result, ensure_ascii=False), and then optionally compress it with gzip.compress and write it to a.json.gz file. Configuration module (7):

[0017] Save the key parameters of pymupdf and camelot in the form of a JSON file, such as "stream_edge_tol": 500, for easy parameter adjustment without code in the later stage. Exception handling module (8):

[0018] Wrap a try / except around the system entry to catch exceptions such as FileNotFoundError, pdfium.PdfError, camelot.CamelotError, etc., and write the exception type, timestamp, and file path to error.log for easy operation and maintenance positioning.

[0019] Through the above steps, this embodiment can complete the structured extraction of text, tables, and headings from any PDF file that conforms to the layout specification at one time, and save it as a compressed JSON file for direct invocation by subsequent business systems.

Claims

1. A PDF file structured data extraction system, characterized in that, Including: A PDF parsing module for reading a PDF file through the PyMuPDF library of Python and outputting page objects; a text extraction module for calling the get_text method of the page object to obtain a plain text string; A table extraction module for identifying and parsing tables in a PDF through the camelot library of Python to output structured table data; A title recognition module for matching a plain text string based on a regular expression to locate first-level titles, second-level titles, and corresponding page numbers; a paragraph splitting module for splitting the plain text string into multiple paragraphs according to the title positions output by the title recognition module; a JSON serialization module for encoding paragraph text, structured table data, and title information into a JSON string and writing it to a disk file; a configuration module for storing initialization parameters of the PyMuPDF library and the camelot library; an exception handling module for capturing exceptions thrown during the execution of the PDF parsing module, the text extraction module, the table extraction module, and the title recognition module and logging them.

2. The PDF file structured data extraction system according to claim 1, characterized in that, Before reading the PDF file, the PDF parsing module verifies whether the file extension is ".pdf". If the verification fails, the process is terminated and an error code is sent to the exception handling module.

3. The PDF dossier structured data extraction system according to claim 1, characterized in that, Before parsing the table, the table extraction module calls the Stream method of the camelot library and sets the flavor parameter to "stream".

4. The PDF dossier structured data extraction system according to claim 1, characterized in that, The regular expression used by the title recognition module is r'^第[一二三四五六七八九十]+章\s+.+$'.

5. A method for extracting structured data from PDF dossiers, characterized in that, Including the following steps: Opening a PDF file through the PyMuPDF library of Python and obtaining a list of page objects; traversing the list of page objects, calling the get_text method for each page to extract a plain text string; concatenating the plain text strings into a full-text string; using the camelot library to perform table parsing on the PDF file to generate a list of DataFrame objects; matching titles in the full-text string based on a regular expression and recording the title line numbers; splitting the full-text string into multiple paragraphs according to the title line numbers; serializing the paragraph text, DataFrame objects, and title information into JSON format; writing the JSON data to a disk file.

6. The method for extracting structured data from PDF dossiers according to claim 5, characterized in that, Before opening the PDF file, check whether the file path exists. If the path does not exist, raise a FileNotFoundError exception.

7. The method for extracting structured data from PDF dossiers according to claim 5, characterized in that, Before calling the camelot library, convert the PDF file to a temporary image file and set the image resolution to 300 dpi.

8. The method for extracting structured data from PDF dossiers according to claim 5, characterized in that, Before serializing JSON, perform null value filtering on the DataFrame object and delete all cells with a value of NaN.

9. The method for extracting structured data from PDF dossiers according to claim 5, characterized in that, ​

Images