A method, apparatus, device, and medium for intelligent parsing of unstructured PDF documents

Abstract

This application discloses an intelligent parsing method, apparatus, device, and medium for unstructured PDF documents, relating to the field of document parsing. The method includes: acquiring the PDF document to be parsed, parsing the page elements in the PDF document, and generating a document metadata dictionary; if the PDF document does not contain extractable text, converting the PDF document into an image and performing optical character recognition to generate first structured data; if the PDF document contains extractable text, determining whether the PDF document contains a table; if the PDF document does not contain a table, using PDFMiner to extract the text and generating second structured data; if the PDF document contains a table, performing multimodal feature extraction and feature fusion on the PDF document according to the document metadata dictionary to obtain multimodal fusion features, and generating third structured data based on the multimodal fusion features. This application improves the parsing accuracy and efficiency of PDF documents.

Description

Technical Field

[0001] This application relates to the field of document parsing, and in particular to a method, apparatus, device, and medium for intelligent parsing of unstructured PDF documents. Background Technology

[0002] The common parsing methods for Portable Document Format (PDF) documents include: (1) Text extraction technology: pure text parsing based on PDFMiner, which is suitable for PDFs where the text can be directly extracted, but is powerless for scanned documents or image-based text; (2) Optical Character Recognition (OCR) technology: optical character recognition based on Tesseract, which is suitable for scanned documents, but is insufficient in terms of segmentation and semantic understanding of text blocks with complex layouts; (3) Layout analysis technology: document layout detection based on deep learning models (such as YOLOX), which can identify text boxes, images, tables and other areas, but lacks a multimodal decision fusion mechanism.

[0003] In addition to the above problems, the existing technology also has the following defects: (1) Single-modal processing limitations: Traditional PDF parsing methods rely only on text extraction or single OCR technology. When faced with complex layouts (such as charts, formulas, and images embedded in text), information omissions or structural errors often occur; (2) Single strategy: Existing systems lack dynamic decision-making mechanisms and cannot automatically select the optimal parsing strategy based on document characteristics (such as text extractability and image complexity), resulting in limited parsing efficiency and accuracy; (3) Insufficient multimodal fusion: It fails to achieve deep fusion of multimodal information such as text semantics, image layout, and coordinate positioning, making it difficult to accurately understand the logical structure of the document (such as tables, lists, and paragraph levels). Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, device, and medium for intelligent parsing of unstructured PDF documents, which can improve the parsing accuracy and efficiency of PDF documents.

[0005] To achieve the above objectives, this application provides the following solution:

[0006] Firstly, this application provides a method for intelligent parsing of unstructured PDF documents, including:

[0007] Obtain the PDF document to be parsed, parse the page elements in the PDF document, and generate a document metadata dictionary; the document metadata dictionary includes the number of pages and the distribution of elements;

[0008] Based on the document metadata dictionary, determine whether the PDF document contains extractable text;

[0009] If the PDF document does not contain extractable text, the PDF document is converted into an image and optical character recognition is performed to generate the first structured data.

[0010] If the PDF document contains extractable text, then traditional visual methods and deep learning algorithms are used to determine whether the PDF document contains a table;

[0011] If the PDF document does not contain tables, PDFMiner is used to extract the text and generate the second structured data.

[0012] If the PDF document contains tables, then multimodal feature extraction and feature fusion are performed on the PDF document according to the document metadata dictionary to obtain multimodal fusion features, and third structured data is generated based on the multimodal fusion features; wherein, the first structured data, the second structured data, or the third structured data are the parsing results of the PDF document.

[0013] Secondly, this application provides an intelligent parsing device for unstructured PDF documents, comprising:

[0014] The preprocessing module is used to obtain the PDF document to be parsed, parse the page elements in the PDF document, and generate a document metadata dictionary; the document metadata dictionary includes the number of pages and the distribution of elements;

[0015] The text determination module is used to determine whether the PDF document contains extractable text based on the document metadata dictionary;

[0016] The first parsing module is used to convert the PDF document into an image and perform optical character recognition to generate first structured data when the PDF document does not contain extractable text.

[0017] The table determination module is used to determine whether the PDF document contains tables when the PDF document contains extractable text, using traditional visual methods and deep learning algorithms.

[0018] The second parsing module is used to extract text using PDFMiner and generate second structured data when the PDF document does not contain tables.

[0019] The third parsing module is used to extract and fuse multimodal features of the PDF document according to the document metadata dictionary when the PDF document contains tables, to obtain multimodal fused features, and to generate third structured data according to the multimodal fused features; wherein the first structured data, the second structured data, or the third structured data are the parsing results of the PDF document.

[0020] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described intelligent parsing method for unstructured PDF documents.

[0021] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described intelligent parsing method for unstructured PDF documents.

[0022] According to the specific embodiments provided in this application, this application has the following technical effects:

[0023] This application provides a method, apparatus, device, and medium for intelligent parsing of unstructured PDF documents. By determining whether the PDF document contains extractable text and whether it contains tables, different parsing strategies are selected, multimodal data such as text and images are fused, and a dynamic strategy selection mechanism is combined to achieve high-precision parsing and structured output of unstructured documents. This solves the problems of single-modal processing, rigid strategies, and insufficient multimodal fusion in the prior art, and improves the parsing accuracy and efficiency of PDF documents. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the overall process of an intelligent parsing method for unstructured PDF documents provided in an embodiment of this application.

[0026] Figure 2 This is a detailed flowchart illustrating an intelligent parsing method for unstructured PDF documents provided in an embodiment of this application.

[0027] Figure 3 This is a schematic diagram of the functional modules of an intelligent PDF document parsing device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0029] The purpose of this application is to integrate multimodal data such as text and images, and combine them with a dynamic strategy selection mechanism to achieve high-precision parsing and structured output of unstructured documents, thereby solving the problems of single-modal processing, rigid strategies and insufficient multimodal fusion in the existing technology. Furthermore, it automatically selects the best parsing strategy based on the document type (text extractable / scanned document) and complexity (whether it contains tables).

[0030] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] First, some technical terms involved in the embodiments of this application will be introduced.

[0032] 1. PDFMiner is a Python library focused on accurately extracting text, images, tables, and other data from PDF files. It is particularly adept at handling complex document formats such as academic papers, contracts, and reports, and can accurately restore the content structure and semantic information.

[0033] 2. pdf2image is a Python library that can convert PDF files to image formats such as PNG and JPEG. It is commonly used to extract images from PDFs, process scanned documents, or as an OCR preprocessing tool. It leverages underlying PDF rendering tools (such as Poppler) to achieve the conversion, rendering each page of a PDF as a high-resolution image, and provides a simple and easy-to-use API interface.

[0034] 3. YOLOX is an advanced version of the YOLO series, developed by the Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences in 2021. While maintaining YOLO's high-speed detection advantages, it introduces innovative technologies to significantly improve detection accuracy, especially performing well in small targets and dense scenes. It is an open-source target detection algorithm.

[0035] 4. Bidirectional Encoder Representations from Transformers (BERT) is a pre-trained language model developed by Google in 2018. Based on the Transformer architecture, it significantly improves the performance of Natural Language Processing (NLP) tasks through bidirectional context learning technology, ushering in a new era of "pre-training + fine-tuning" in NLP and has become the basic model for various NLP applications.

[0036] 5. spaCy is an industrial-grade NLP library developed using Python and Cython, capable of efficiently processing and analyzing large amounts of text data. It provides pre-trained models, multi-language support, and a concise Application Programming Interface (API), enabling rapid implementation of common NLP tasks such as word segmentation, part-of-speech tagging, named entity recognition (NER), and dependency parsing. It is widely used in information extraction, text classification, and semantic understanding.

[0037] 6. OCR is a technology that converts text in an image into machine-readable text. It uses computer vision and machine learning algorithms to recognize text (such as printed or handwritten text) in an image and convert it into text format. It is widely used in scenarios such as document digitization, information extraction, and image translation.

[0038] 7. Tesseract OCR is an open-source OCR engine that provides a Python interface and can convert text in images into machine-readable text format. It supports over 100 languages, including Chinese, English, Japanese, and Korean, and has wide applications in document scanning, image translation, license plate recognition, digital libraries, and other fields.

[0039] In one exemplary embodiment, such as Figure 1 and Figure 2 As shown, a method for intelligent parsing of PDF documents is provided. This method is executed by a computer device and includes steps 1 to 6.

[0040] This application first designs four strategy patterns.

[0041] (1) auto: Automatic strategy. Based on whether the text extraction is successful or not, define the flag variable pdf_text_extractable. Based on whether the table recognition is successful or not, define the flag variable infer_table_structure. Based on the positive or negative of the two variables, dynamically select the parsing strategy.

[0042] (2) ocr_only: Pure OCR extraction strategy, execute step 3. Convert the PDF page into an image (e.g., 300 DPI) using pdf2image, without performing block segmentation or layout analysis. Call Tesseract to perform OCR recognition on the entire page image and generate a text stream. Element classification: Only generate Text elements, without performing table or hierarchical analysis (due to lack of layout information). Finally, generate a structured output containing only Text elements.

[0043] (3) fast_text_extract: A fast text extraction strategy that executes step 5. It skips complex steps such as image segmentation and YOLOX detection, and directly extracts searchable text using PDFMiner. Text cleaning: Removes redundant spaces and line breaks, and standardizes the format using regular expressions (e.g., re.sub(r'[\n\t\r]+','')). Hierarchical marking: Based on font size (e.g., ≥18pt for Title) and spaCy paragraph segmentation, it constructs a simple hierarchy (Title→Body), ultimately generating a lightweight structured output that focuses on text content and a simple hierarchy.

[0044] (4) layout_resolve: High-precision layout analysis + OCR strategy. Execute step 6, infer the table row and column structure based on the fusion features, generate an HTML table with semantic tags, and finally generate a JSON / XML output with a complete table structure.

[0045] In summary, the detailed principles of the four strategies are shown in Table 1.

[0046] Table 1 Four Strategies

[0047]

[0048] The four strategies cover more than 95% of document types through a dynamic decision-making mechanism, achieving "quick parsing of simple documents and accurate parsing of complex documents", and ultimately outputting standardized structured data to support intelligent applications in finance, law and other fields.

[0049] In practical applications, PDF parsing can be performed by specifying a strategy or by following the default process and dynamically selecting a parsing strategy, which can be adapted to different application scenarios.

[0050] Step 1: Obtain the PDF document to be parsed, parse the page elements in the PDF document, and generate a document metadata dictionary. The document metadata dictionary includes the number of pages and the distribution of elements.

[0051] In a specific application example, the `pdfminer.high_level.extract_pages()` function in Python is called to parse the PDF document page by page, determining the number of pages and the distribution of elements to generate a document metadata dictionary. Specifically, a user-uploaded PDF document, assuming it is "2024 Q3 Financial Report.pdf", is loaded through the file input interface (pdfminer library). `pdfminer.high_level.extract_pages()` is then called to parse the PDF document page by page, obtaining the types and coordinates of elements such as text boxes, images, and tables, and generating a document metadata dictionary containing the number of pages and the distribution of elements.

[0052] Step 2: Based on the document metadata dictionary, determine whether the PDF document contains extractable text.

[0053] In a specific application example, based on the document metadata dictionary, the pdfminer.high_level.extract_text() function in Python is called to extract the text stream. If the text stream is successfully extracted, the PDF document contains extractable text; otherwise, the PDF document does not contain extractable text.

[0054] Step 3: If the PDF document does not contain extractable text, the PDF document is converted into an image and optical character recognition is performed to generate the first structured data.

[0055] Specifically, if the pdfminer.high_level.extract_text() function returns empty text, meaning the PDF document does not contain extractable text, then pdf_text_extractable=False is assigned, and the ocr_only strategy is directly triggered for parsing.

[0056] Step 4: If the PDF document contains extractable text, then traditional visual methods and deep learning algorithms are used to determine whether the PDF document contains a table.

[0057] Furthermore, if the PDF document contains extractable text (i.e., content such as cover title, body paragraphs, etc.), then pdf_text_extractable=True is assigned, and text cleaning and preprocessing are performed, as in steps 41 to 43 below.

[0058] Step 41: Standardize the format by removing whitespace characters from the text using the regular expression re.sub(r'[\n\t\r]+', ' '), such as converting "Cash and cash funds\n\t\t50 million yuan" to "Cash and cash funds 50 million yuan", thus improving the readability of the text.

[0059] In regular expressions, `re.sub` is a Python function used to replace matched content in a string. `r'[\n\t\r]+'` is a regular expression used to match specific whitespace characters; `r` represents the raw string, avoiding escaping backslashes, and `[...]` is a character set that matches any single character within the square brackets. " is a newline character, "\t" is a tab character, "\r" is a carriage return character, and "+" indicates that the preceding character or character set can appear once or multiple times. "' '" (space) is the replacement content of the `re.sub` function, meaning that the matched whitespace character will be replaced with a single space. Using the regular expressions above, various whitespace characters in a string can be effectively replaced with a single space, thus cleaning up redundant whitespace in the text.

[0060] Step 42, paragraph segmentation: use spaCy's en_core_web_sm model to segment the main text paragraphs according to semantic boundaries (e.g., "Important Accounting Policy Explanation" is segmented into 3 natural paragraphs: "Important", "Accounting", and "Policy Explanation"), to build the logical hierarchy foundation.

[0061] Among them, en_core_web_sm is a small English language model provided by spaCy, specifically designed for NLP tasks. It is a pre-trained model based on deep learning technology, capable of handling a variety of common NLP tasks, such as word segmentation, part-of-speech tagging, dependency parsing, and named entity recognition (NER).

[0062] Step 43, special tag processing, using regular expressions Identify footnotes (such as " Extract the data as of September 30, 2024, and use the coordinates extracted by PDFMiner (such as the Y-axis being greater than 80% of the page height) to mark "Data as of September 30, 2024" as a footer type.

[0063] Through the text cleaning and preprocessing steps 41 to 43, this application can achieve the following three beneficial effects.

[0064] (1) Improve the accuracy of parsing: Standardized formatting eliminates text formatting interference, making subsequent understanding and processing of text content more accurate; paragraph segmentation reasonably divides text into logical units, and special markings mark key information, which helps to fully and accurately understand the document content, reduce parsing errors caused by improper text processing, and improve the overall parsing quality.

[0065] (2) Text cleaning and preprocessing can lay the foundation for subsequent multimodal processing, feature fusion, structured output and other steps, and ensure the smooth progress of the entire parsing process. If these preprocessing steps are missing, subsequent text-based analysis and processing will face problems such as data chaos and semantic ambiguity, which will affect the normal operation of the parsing system and the final result.

[0066] (3) Enhanced parsing adaptability: Through standardized processing, paragraph segmentation and special marking of various formats and types of text, it can adapt to diverse PDF documents. Whether it is academic papers, financial reports or legal contracts, it can effectively process the text content, thus expanding the scope of application and applicability.

[0067] In a specific application example, step 4, which uses traditional visual methods and deep learning algorithms to determine whether the PDF document contains a table, specifically includes steps 44 to 46.

[0068] Step 44: Use the OpenCV edge detection algorithm and Hough transform to filter the horizontal lines in the PDF document and calculate the horizontal line density (e.g., 8 horizontal lines are detected in a 100×100 pixel grid).

[0069] Step 45: A deep learning algorithm is used to perform object detection on the PDF document to determine the table detection results. The table detection results include table bounding boxes, confidence scores, and category labels. The deep learning algorithm used is the YOLOX model.

[0070] This application calls a lightweight model (such as yolox-nano.onnx) to detect PDF documents and outputs table bounding boxes (such as x1=150, y1=300, x2=550, y2=500), confidence score (0.92) and category labels, including Title, Table, Footnote, etc.

[0071] Step 46: Based on the horizontal line density and the table detection results, determine whether the PDF document contains a table.

[0072] Specifically, if the horizontal line density is ≥4 lines / 100 square pixels and the confidence level of the YOLOX model detection is ≥0.8, it is determined that a high-confidence table exists, and infer_table_structure=True is assigned, and the layout_resolve strategy is enabled.

[0073] Step 5: If the PDF document does not contain tables, then use PDFMiner to extract text (i.e., execute the fast_text_extract strategy) to generate the second structured data.

[0074] Step 6: If the PDF document contains tables, then based on the document metadata dictionary, perform multimodal feature extraction and feature fusion on the PDF document to obtain multimodal fused features, and generate third structured data based on the multimodal fused features. The first structured data, the second structured data, or the third structured data are the parsing results of the PDF document.

[0075] In a specific application example, step 6 involves extracting and fusing multimodal features from the PDF document based on the document metadata dictionary to obtain multimodal fused features, including steps 61 to 65.

[0076] Step 61: Based on the document metadata dictionary, perform text semantic tagging, image block calibration, and coordinate calibration on the PDF document to obtain preprocessed text semantic data, preprocessed image data, and preprocessed coordinate data.

[0077] (1) Text Semantic Tagging: Entities in the preprocessed text semantic data are extracted using spaCy (e.g., "monetary funds" is tagged as FINANCIAL_ENTITY), and font sizes are extracted using the LTChar.size attribute of PDFMiner (e.g., ≥18pt is tagged as Title, 12pt as Body), constructing a Title→Body hierarchical relationship. The hierarchical relationship is constructed based on three constraints: font size, spatial location, and semantic association, forming a logical chain from title to body text. Specifically, font size is the primary indicator, as shown in Table 2, with spatial location used as a supplementary verification method.

[0078] Table 2 Font Size

[0079]

[0080] The title positioning rules are as follows: in the point space of the PDF document, the Y-axis coordinate of the title must be less than 20% of the page height (this can be adjusted according to the document type, such as 15% for academic papers and 20% for reports), and the X-axis coordinate of the title must be located in the middle area of ​​the page width, with a deviation not exceeding 10% of the page width. Through multimodal collaboration of "spatial location + semantic features + visual positioning," the title positioning rule becomes a key link connecting the physical layout and logical semantics of the document. Its accuracy directly affects the quality of all subsequent parsing steps and is one of the fundamental core technologies for achieving intelligent parsing of unstructured PDFs.

[0081] In a specific example, the pixel-to-point conversion process is as follows: Given DPI = 300, 1 point (pt) = Since pixels are pixels, the formula for converting pixel coordinates to point coordinates is: DPI stands for pixels per inch.

[0082] The process of calculating page height is as follows: If the page height is 600 pixels, the formula for converting it to point space is: .

[0083] The conversion process for Y-axis coordinates is as follows: If the Y-axis pixel coordinate of the Title is 120 pixels, the formula for converting it to point coordinates is: .

[0084] The threshold determination process is as follows: 20% of the page height is... At this point, the Y-axis coordinate of Title is 28.8, which meets the title positioning conditions.

[0085] Title positioning serves as the starting point for constructing the document's logical hierarchy. By determining the root node of each chapter through title positioning, and combining font size (e.g., ≥18pt) and semantic tags (e.g., the keyword "balance sheet" extracted by BERT), a "Title→Body" hierarchy is built. For example, after positioning the title "2024 Q3 Financial Report," paragraphs with a vertical distance of less than 50 pixels below it are automatically assigned to the body text, forming a logical chain. Further, the accuracy of the hierarchical relationship is ensured through triple verification using visual positioning of the title (the title area detected by YOLOX), textual semantics (BERT vector), and coordinate space (the top area of ​​the Y-axis).

[0086] Furthermore, if the vertical distance between the Title and the subsequent table is less than 50 pixels, a "Title→Table" association is established through the parent field. The attention score of the Title area is typically >0.9, which guides OCR to prioritize parsing title-related content, improving the efficiency of extracting key information. If the Title text is misidentified by OCR as "2024 Q3 Financial Announcement," and the cosine similarity between the BERT vector and "financial report" is >0.85, combined with the Title positioning rule (top of Y-axis + center of X-axis), dynamic weight correction (dominated by visual weight 0.6) is triggered, correcting it to the correct text. A unified Title positioning rule ensures the standardization of the hierarchical structure of different documents, facilitating batch processing and retrieval (such as quickly locating the financial report section through title keywords).

[0087] This application utilizes a logical hierarchy constructed through Title positioning to improve document element classification accuracy to over 95%, avoiding the hierarchical confusion issues inherent in traditional single-modal parsing. The Title, as the root node, supports the hierarchy field in JSON / XML output (e.g., {"id":"Title_001","children":["Table_001"]}), adapting to the intelligent retrieval needs of fields such as finance and law. In financial report analysis, it can quickly locate the "Balance Sheet" title, link subsequent table data, and automatically extract key values ​​such as "Cash and Cash Equivalents," reducing manual data entry errors. In legal contract review, the Title is used to locate the "Breach of Contract" section and link attached charts, improving clause retrieval efficiency by over 30%. In digital archive management, OCR recognition and positioning of the Title in scanned documents create a searchable electronic catalog, achieving automated archive classification.

[0088] The positioning rules for the Body are: immediately below the Title, with a vertical distance of less than 50 pixels and a font size of 12pt. The logical relationship is verified using spaCy entity recognition results: if the Title contains a FINANCIAL_ENTITY entity (such as "Balance Sheet"), then the Body paragraph below it must contain a similar entity (such as "Cash and Cash Equivalents" or "Beginning Balance"); otherwise, a hierarchy conflict check is triggered. Example: The paragraph below the title "Balance Sheet" (FINANCIAL_ENTITY) contains "Cash and Cash Equivalents" (FINANCIAL_ENTITY), strengthening the Title→Body association; if a "Company Logo" (a non-financial entity) appears, it is determined to be an independent image element and is not included in the Body hierarchy.

[0089] (2) Image block calibration: First, the pdf2image function is called to convert the page containing the table into a 300 DPI image (800×600 pixels). Then, the image is divided into 10×10 pixel sub-blocks (4800 blocks in total) by sliding window. After that, perspective transformation (cv2.findHomography) is applied for correction and tilting (the error is reduced from 15 to <1°).

[0090] In PDF document parsing, chunking refers to the process of dividing a PDF page into multiple small pixel sub-blocks along both the horizontal and vertical axes after it has been converted into an image. Horizontal chunking refers to segmenting along the width of the image, and the number of sub-blocks obtained is called the horizontal chunk number. Vertical chunking refers to segmenting along the height of the image, and the number of sub-blocks obtained is called the vertical chunk number. The total number of chunks is the product of the horizontal chunk number and the vertical chunk number.

[0091] Taking an image 800 pixels wide and 600 pixels high as an example, if it is divided into sub-blocks of 10×10 pixels, then the horizontal number of blocks is: with an image width of 800 pixels and each sub-block width of 10 pixels, it can be divided into 800÷10=80 blocks horizontally; the vertical number of blocks is: with an image height of 600 pixels and each sub-block height of 10 pixels, it can be divided into 600÷10=60 blocks vertically. The total number of blocks is: 80 horizontal blocks × 60 vertical blocks = 4800 small image blocks.

[0092] By segmenting large images into small sub-blocks, the input requirements of lightweight object detection models (such as YOLOX-nano) can be adapted, the computational cost of a single inference can be reduced, small elements (such as footnotes and icons) can be avoided from being ignored in large images, and the ability to capture details can be improved through sub-block level detection.

[0093] After segmentation, each YOLOX inference operation only processes 10×10 pixel sub-blocks, which reduces computational cost compared to full-image detection (800×600 pixels). The resolution speed has been increased from 12.7s / page to 8.3s / page. Smaller sub-blocks prevent targets in large images from being compressed or occluded. For example, a 10×10 pixel sub-block can accurately capture a 5×5 pixel footnote icon, which is easily missed by traditional full-image detection.

[0094] For documents containing multi-column text and nested tables, segmentation allows for segment-by-segment parsing, avoiding layout chaos during full-image parsing. For example, after segmenting the terms and attachments of a legal contract, the visual features of each sub-block are independently detected, and then the overall structure is restored by merging coordinates. For scanned PDFs, segmentation allows for independent OCR of each sub-block, reducing text distortion errors in full-image OCR and improving the parsing accuracy of scanned documents to 87.5%.

[0095] Block segmentation, as a key operation in PDF parsing that "breaks down a whole into parts", reduces computational complexity, improves detection accuracy, and supports spatial alignment. It serves as a bridge connecting image preprocessing and multimodal feature extraction, ultimately ensuring efficient and accurate parsing of complex documents. It is especially indispensable in scenarios with high layout requirements, such as financial reports and legal contracts.

[0096] In one instance, the coordinates of the top-left and bottom-right corners of the sub-block in the i-th row (vertical, starting from 0) and j-th column (horizontal, starting from 0) are: (x1, y1) = (j×10, i×10), (x2, y2) = ((j+1)×10, (i+1)×10). Example: The coordinates of the sub-block in the 1st row and 1st column are (10, 10, 20, 20).

[0097] (3) Coordinate calibration: The rect_to_bbox function is used to convert the coordinates of the PDF point space to the coordinates of the pixel space. The default coordinates of PDF are based on the bottom left corner of the page. When moving to the right (X-axis) and up (Y-axis), the coordinate values ​​gradually increase. However, the coordinates of image processing are based on the top left corner of the page. When moving to the right (X-axis) and down (Y-axis), the coordinate values ​​gradually increase. This is completely opposite to the Y-axis direction of PDF. Therefore, it is necessary to convert the coordinates of PDF points to pixel coordinates.

[0098] First, convert the point coordinates (such as the position of text or images) in the PDF to pixel coordinates, taking into account the document's resolution (DPI). For example, 1 point in a PDF at 300 DPI is approximately equal to 1.25 pixels (because 72 points = 1 inch, 300 pixels = 1 inch, so 1 point ≈ 300 / 72 ≈ 4.167 pixels). The conversion logic is: the larger the X-coordinate value of the PDF, the larger the corresponding X-coordinate value of the pixel, and the direction is consistent. For example, if X = 100 points in the PDF, simply multiply by the pixel / point ratio when converting to pixels. The Y-coordinate of the PDF starts from the bottom left corner, while the pixel coordinate starts from the top left corner. Therefore, it is necessary to "flip" the Y-axis coordinate—subtract the pixel value corresponding to the Y-coordinate of the PDF from the total pixel height of the page to obtain the Y-coordinate in pixel space.

[0099] Then, the orientation and origin need to be corrected: the Y-axis of the PDF grows upwards, while the Y-axis of the pixel space grows downwards. Mathematical operations are needed to "flip" the Y-coordinate of the PDF to the correct orientation in the pixel space. For example, if the Y-coordinate of an element in the PDF is 500 points and the total page height is 842 points, when converting to pixels, the total pixel height needs to be subtracted from the number of pixels corresponding to that value to ensure the element's position in the image is consistent with the PDF.

[0100] The `rect_to_bbox` function eliminates the coordinate system differences between PDFs and images, unifying the positions of all elements to pixel space. This allows text semantics, visual layout, and coordinate positions to be correlated under the same benchmark, providing an accurate spatial positioning foundation for subsequent multimodal detection, error correction, and structured parsing.

[0101] After converting the block coordinates and the text point coordinates extracted by PDFMiner using rect_to_bbox, the positions of text and visual elements can be accurately matched (such as the table area within the sub-block corresponding to the text "balance sheet"), providing a spatial benchmark for the attention score calculation in subsequent step 63. Furthermore, the adjacency relationship of sub-blocks (such as top and bottom, left and right sub-blocks) can help infer the logical hierarchy of elements; for example, titles and tables within vertically adjacent sub-blocks are determined to be related content.

[0102] Step 62: Extract features from the preprocessed text semantic data, the preprocessed image data, and the preprocessed coordinate data respectively to obtain text semantic vector, visual feature vector, and coordinate feature vector.

[0103] In a specific application example, the BERT model is used to extract the text semantic vector from the preprocessed text semantic data. The YOLOX model is used to extract visual features from the preprocessed image data, obtaining a visual feature vector. The preprocessed coordinate data is normalized to obtain a coordinate feature vector.

[0104] (1) Text semantic vector: including semantic vector, keywords and semantic tags. The preprocessed text semantic data is input into the BERT model to generate a 768-dimensional vector, keywords (such as "monetary capital growth") are extracted, and then the cosine similarity between different text vectors is calculated.

[0105] The BERT model, based on the Transformer architecture, transforms input text into a 768-dimensional semantic vector through bidirectional context learning. This vector contains the text's semantic features, contextual information, and relationships between words, providing a deep numerical representation of the text's semantics. For example, for the text "balance sheet," the values ​​of each dimension in the 768-dimensional vector reflect its relevance to semantics such as "financial statements" and "cash flow," as well as its specific meaning within the document's context.

[0106] In subsequent steps, the semantic relevance between texts can be determined by calculating the cosine similarity between the 768-dimensional vectors. For example, in multimodal feature fusion and conflict resolution, comparing the vector similarity between "balance sheet" and the text in the table area helps determine their logical connection and assists in judging whether the table content belongs to that heading. Combined with tools such as spaCy, the 768-dimensional vectors can enhance the accuracy of entity recognition in text (such as "cash and cash equivalents" marked as FINANCIAL_ENTITY), providing support for constructing the semantic structure of the document. When processing scanned text, if OCR recognition errors occur, the semantic vectors generated by BERT can be used as one of the judgment criteria, combined with visual modal information for cross-modal conflict resolution, correcting erroneous text recognition results.

[0107] Based on BERT semantic vectors, combined with TF-IDF or TextRank algorithms, high-frequency or high-weight words (such as "balance sheet" and "beginning balance") in the text are extracted as keywords.

[0108] Named entity recognition is performed using spaCy to label entity types in text (such as financial entities, dates, and locations), and semantic hierarchies are constructed by combining font sizes extracted by PDFMiner (e.g., ≥18pt is labeled as Title).

[0109] (2) Visual feature vector: Call a lightweight YOLOX model (such as yolox-nano.onnx) to infer each pixel sub-block in the preprocessed image data, and output the bounding box coordinates (x1, y1, x2, y2), class label (such as Table, Title, Footnote, Logo) and confidence score (such as 0.92) of each detected target as the visual feature vector of the pixel sub-block. Concatenate the visual feature vectors of each pixel sub-block into a matrix to obtain the final visual feature vector.

[0110] The output of the YOLOX model not only provides key visual information for multimodal fusion, but also directly affects strategy selection and structured parsing results, forming a technical chain of "image preprocessing → visual detection → feature fusion → structured output" to ensure accurate recognition and location of complex elements such as tables and images in documents.

[0111] Furthermore, when the YOLOX model detects specific visual elements (such as tables or title areas), it performs directional OCR recognition on that area. Using the bounding box coordinates output by the YOLOX model, the pixel sub-blocks to be recognized are located. The Tesseract OCR engine is then invoked to perform high-precision recognition of this area, obtaining the text content and confidence score. The OCR-recognized text content (such as "currency value") is combined with the visual modality's region category ("TableTitle"), and cross-modal error correction is performed through semantic constraints (financial dictionary) and coordinate spatial relationships (located in key areas of financial statements). The coordinates of the OCR-recognized text must be consistent with the coordinates of the visual element; for example, the coordinates of "currency value" must be located within the table title area detected by YOLOX, otherwise it may be judged as a misrecognition.

[0112] (3) Coordinate feature vector: including the normalized coordinates of the text block and the table to quantify the spatial position relationship. The coordinates output by YOLOX are based on pixel space. After being transformed by the rect_to_bbox function, they are mapped to the PDF point space coordinates and normalized to the [0,1] interval to facilitate subsequent multimodal feature fusion (such as combining with text coordinates and semantic features).

[0113] Specifically, the pixel coordinates are scaled to the normalized range required by the model input (e.g., YOLOX requires the coordinate range to be [0,1]). The formula is: , ,in, For normalized x-axis coordinates, Here are the normalized y-axis coordinates. The normalized coordinates of the top-left corner of the sub-block are: The normalized coordinates of the bottom right corner of the sub-block are: Example of a result: The normalized coordinates of pixel coordinates (10,10,20,20) are (0.0125,0.0167,0.025,0.0333).

[0114] "Quantifying spatial relationships" refers to converting the relative positions of various elements (such as text blocks, tables, and images) in a PDF document into calculable numerical information. In this way, computers can accurately understand the physical layout between elements, such as "who is above whom," "how close are they to each other," and "who contains whom," thus providing a data foundation for the logical hierarchy analysis of the document, multimodal information fusion, and structured output.

[0115] The specific process of quantifying spatial relationships is as follows.

[0116] 1) Unify and normalize the coordinate system: First, unify the coordinate system in the PDF document to the pixel coordinates of the image (with the top left corner as the origin). For example, point coordinates in the PDF will be converted to pixel coordinates to ensure that the position of all elements is based on the same reference system. Then, scale the pixel coordinates to the [0,1] range so that the positions of page elements of different sizes can be compared uniformly. For example, if an element is 100 pixels horizontally on the page and the total width of the page is 800 pixels, its normalized horizontal coordinate is 100 / 800 = 0.125.

[0117] 2) Calculate the distance between elements: Using normalized coordinates, calculate the distance between two elements in the vertical direction (Y-axis) and the horizontal direction (X-axis). For example, if the vertical coordinate range of element A is 0.2~0.3 and the vertical coordinate range of element B is 0.35~0.4, then the vertical distance between element A and element B is 0.35-0.3=0.05. A distance threshold can be set (e.g., a vertical distance less than 0.08 corresponds to 50 pixels). When the distance between two elements is less than the threshold, they may be considered logically related (e.g., the title and the table below).

[0118] 3) Determine the positional relationship of elements: If the right-side normalized coordinate of element A is less than the left-side normalized coordinate of element B, then A is to the left of B. For example, if the horizontal coordinate of element A is 0.1~0.3 and the horizontal coordinate of element B is 0.4~0.6, then element A is to the left of element B. If the lower-side normalized coordinate of element A is less than the upper-side normalized coordinate of element B, then element A is above element B. For example, if the vertical coordinate of element A is 0.2~0.3 and the vertical coordinate of element B is 0.4~0.5, then element A is above element B.

[0119] 4) Determine the inclusion relationship of elements: If the coordinate range of element A completely includes the coordinate range of element B (i.e., the left boundary of A ≤ the left boundary of B, the right boundary of A ≥ the right boundary of B, the top boundary of A ≤ the top boundary of B, and the bottom boundary of A ≥ the bottom boundary of B), then element A contains element B. For example, if the coordinate range of a large table includes the coordinates of one of its cells, it means that the cell is part of the table.

[0120] This application quantifies the positional relationships of elements such as text, tables, and images to correlate semantic information with visual layout. For example, the text "Balance Sheet" and the table on the right are spatially adjacent, suggesting a content-related relationship and improving the accuracy of multimodal fusion. Furthermore, logical relationships are inferred based on the distance and orientation between elements. For instance, the title and the table 28 pixels below it are considered parent-child relationships ("title-table") because the distance is less than 50 pixels; the arrangement of multi-column text from left to right restores the reading order. The quantified coordinate information accurately records the position of elements in the document (with an error of less than 5 pixels). This data is appended as metadata to the structured output (such as JSON) to support subsequent visual verification and document reconstruction. For example, the parsed table coordinates can be directly used for page rendering, ensuring consistency with the original PDF layout.

[0121] Step 63: A three-modal cross-attention mechanism is adopted to perform cross-modal attention fusion on the text semantic vector, the visual feature vector and the coordinate feature vector to quantify the semantic-spatial correlation between the text and the visual elements.

[0122] The formula for the trimodal cross-attention mechanism is as follows: The meanings and examples of each parameter are shown in Table 3.

[0123] Table 3. Parameter Meanings and Examples

[0124]

[0125] In step 63, the visual feature vector and the coordinate feature vector are first concatenated along their dimensions to form a joint feature matrix. Then, the semantic vector of the text is multiplied by the transpose of the joint feature matrix to calculate the initial relevance (e.g., the dot product of the text "balance sheet" and the table features is 0.72). This indicates that the semantic-spatial matching degree between the text and the table region reaches 72%. The relevance is then normalized to an attention score using the Softmax function, increasing the attention score of the table region to 0.91. This guides the OCR engine to prioritize processing this highly relevant region, avoiding redundant calculations from full-image scanning.

[0126] By calculating the attention scores of the text semantic vector and the visual feature vector, the degree of correlation between the two is quantified (value ∈ [0,1]), and highly correlated regions are processed first. For example, the attention score between the text "balance sheet" and the table region detected by YOLOX is 0.91, indicating that the semantic-spatial matching degree between the two is extremely high. Therefore, OCR parsing of the table region is prioritized to avoid redundant calculations of full-image scanning and improve efficiency by 30%.

[0127] Attention score, as a weighting criterion for multimodal fusion, can be used to adjust the contribution of text, visual, and coordinate modalities. For example, when there is a conflict between the OCR recognition result and the visual localization (such as misidentification of "currency funds"), if the attention score of the visual region is higher (such as 0.92), the visual modal weight (0.6) will dominate the error correction, and the result will be corrected to "currency funds" in combination with the financial dictionary, improving the text accuracy by 12.5%.

[0128] Attention scores guide the OCR engine to prioritize high-relevance regions through cross-modal correlation quantization and dynamic region priority sorting. The specific mechanism and processing logic are as follows.

[0129] (1) The correlation between the text and each visual element is calculated by performing a dot product operation on the text semantic vector, visual feature vector, and coordinate feature vector. The attention score is obtained after Softmax normalization. Example: The BERT semantic vector of the text "balance sheet" is dot-producted with the table region features detected by YOLOX. The initial correlation is 0.72. After Softmax, the attention score is increased to 0.91, indicating that the semantic-spatial matching degree between the text and the table region is extremely high.

[0130] (2) The OCR engine (such as Tesseract) sorts the visual element regions from high to low according to the attention score, and prioritizes the parsing of high-score regions (such as the table title area with an attention score > 0.8) to avoid redundant calculations of full-image scanning.

[0131] (3) Use refined OCR (such as 300DPI image segmentation and perspective correction) for high-score areas, and use fast extraction or skipping strategies for low-score areas (such as background images with attention scores <0.5).

[0132] Specifically, the processing flow for highly relevant regions is as follows: An attention score is generated for each visual element using a trimodal cross-attention mechanism. Elements are then arranged in descending order of score to form a parsing queue (e.g., table → title → footnote). Tesseract is used to target high-score regions for directional recognition, and semantic constraints (e.g., a financial dictionary) are applied to improve accuracy. Example: In parsing a financial report table, the attention score between the table title "Balance Sheet" and the table area is 0.91. OCR prioritizes parsing this area, extracting the table structure and data using the layout_resolve strategy, improving accuracy to 95.1%. An HTML table with semantic tags is generated, and coordinate metadata is recorded.

[0133] The processing flow for low-relevance regions is as follows: If the region's attention score is <0.5 and text extractability is True (pdf_text_extractable=True), then PDFMiner is used to quickly extract the text, skipping OCR. This reduces processing time from 12.7s / page to 8.3s / page, a 34.6% efficiency improvement. If the region is a scanned document with a low attention score, pure OCR is used to extract the text, but no detailed layout analysis is performed. This serves as a fallback solution covering over 95% of document types. Example: Processing the background logo region. The logo region has an attention score of 0.23, indicating it is non-critical content. Only image coordinates and category are extracted; OCR text recognition is not performed, reducing unnecessary computation and focusing on core data analysis.

[0134] By dynamically allocating computing resources based on attention scores, high-relevance areas are parsed with fine detail (95.1% accuracy), while low-relevance areas are processed quickly (reducing processing time by 34.6%). Furthermore, the strategy automatically switches based on the attention score without manual intervention, adapting to over 95% of document types, including text, scanned documents, and complex layouts. Attention scores are recorded as metadata in the JSON output, supporting result verification and strategy optimization. Through semantic-spatial dual correlation metric measurement, attention scores enable intelligent scheduling of the OCR engine, resolving the efficiency bottlenecks of traditional full-image parsing and the parsing errors in complex layouts, making it particularly suitable for scenarios with high document parsing accuracy requirements, such as finance and law.

[0135] Step 64: Based on the semantic-spatial correlation degree, the text semantic vector, the visual feature vector, and the coordinate feature vector, a pre-trained neural network model is used to dynamically adjust the weights of text, vision, and coordinates to obtain text weight, visual weight, and coordinate weight.

[0136] The neural network model is an error correction network, a lightweight neural network used to calculate modal weights during multimodal feature fusion, resolving conflicts between text, visual, and coordinate features (such as semantic inconsistencies between OCR recognition errors and visual localization). The error correction network is a fully connected neural network, with a simple and efficient structure. It does not require complex convolutional or recurrent structures, directly processes vector inputs and weight outputs, and can adaptively learn weights.

[0137] The principle behind error correction networks in achieving error correction is as follows.

[0138] (1) Multimodal evidence fusion: When there is a contradiction between the text modality and the visual / coordinate modality (e.g., the semantics of the OCR result are unreasonable but the visual localization is clear), the error correction network resolves the conflict through weight adjustment. The high-confidence modality is given higher weight, for example: visual modality confidence 0.92 > text modality confidence 0.65 → assigned higher weight. The coordinate modality verifies the spatial proximity of the text and the table → assigned weight to strengthen the visual decision.

[0139] (2) Semantic constraints and dynamic adaptation: Domain-specific dictionaries (such as financial terms) are incorporated during network training. When the edit distance between the OCR result and the dictionary item is ≤1 and the visual weight is >0.5, error correction logic is triggered. The weights are dynamically adjusted according to the document area type. For example, for table title areas: the visual weight is automatically increased (≥0.5), and the text weight is decreased (≤0.4), because the visual positioning of the table title (such as borders and positions) is more reliable than the text content.

[0140] The regions include: table title regions, body text regions, footnote regions, header / footer regions, image regions, and list regions. The visual modality weight is higher in the table title region because the position of the table title (e.g., fixed at the top of the table) and visual features (e.g., larger font, borders) are easier for the YOLOX model to accurately detect than text content, resulting in higher reliability. In the body text region, the text and visual modal weights are relatively balanced (e.g., 0.5:0.5) because the content relies on both text semantic understanding and visual layout (e.g., paragraph spacing, font style). In the footnote region, the coordinate modal weight increases because the spatial position of the footnote (bottom of the page) is the primary recognition criterion, while the importance of visual and text modalities decreases relatively. In the image region, the visual modal weight dominates, while the text modal weight is lower because the type of image elements (e.g., logos, charts) is mainly determined through visual detection.

[0141] (3) End-to-end training optimization: The cross-entropy loss function is adopted, with the difference between the parsed results and the manual annotation as the optimization target. The weight matrix is ​​optimized through the backpropagation algorithm, so that the model prioritizes the high-confidence modality in conflict scenarios, thereby improving the error correction accuracy. For example, after training, the text accuracy in the OCR error correction scenario increased from 87.5% to 100%, and the confidence increased from 0.65 to 0.812.

[0142] Among them, the parsing result is the text content, confidence score, element category, and relevant metadata output by the error correction network through multi-modal fusion and error correction. Among them, the text content is the text recognized by OCR and corrected through cross-modal error correction (such as correcting "currency posture fund" to "currency fund"). The confidence score is the reliability score of the result after fusing multi-modal information. The relevant metadata includes metadata such as the spatial coordinates and hierarchical relationships of the elements (such as the parent-child element IDs). Annotation refers to the correct text, element type, and hierarchical relationship generated manually or through a standard process, which serves as the supervision signal for model training, including: the correct text label (the accurate text content in the original document), the true element category, and the true hierarchical relationship (the logical association between elements).

[0143] Traditional OCR only relies on image features and is vulnerable to font and noise effects (such as misrecognizing "资" as "姿"). The error correction network realizes cross-modal collaborative error correction by fusing visual positioning (table title area) and coordinate proximity (spatial association with the table), combined with semantic constraints (financial dictionary). In the OCR recognition and error correction case, when Tesseract misrecognizes "currency fund" as "currency posture fund", the error correction network outputs weights, and the visual modality weight is significantly higher than the text modality. Finally, the fused feature is mainly dominated by 60% visual features, combined with the semantic constraints of the table title, and is automatically corrected to "currency fund". This fused feature not only corrects the error but also improves the confidence score, from 0.65 to 0.812, enhancing the consistency of text with visual and coordinate information.

[0144] In documents with complex layouts such as financial reports and legal contracts, the error correction network can ensure that the accuracy of table data extraction is increased from 65.2% to 95.1%, and the parsing time of complex documents is reduced by 34.6% (from 12.7 s / page to 8.3 s / page). For example, under the action of the error correction network, "currency posture fund" is corrected to "currency fund" through the dominance of visual weight and semantic dictionary matching, and the confidence after fusion is increased by 24.9% (from 0.65 to 0.812).

[0145] In summary, the error correction network maps multi-modal features into dynamic weights through a fully connected neural network structure, realizing an error correction mechanism of "dominance by high-confidence modalities and collaboration of multi-modal evidence", effectively solving the error problem of traditional single-modal parsing, which is one of the core innovation points for improving the parsing accuracy of unstructured PDFs.

[0146] Step 65, according to the text weight, the visual weight, and the coordinate weight, perform weighted summation on the text semantic vector, the visual feature vector, and the coordinate feature vector to obtain a multi-modal fusion feature.

[0147] Specifically, use the formula and Multimodal fusion features are obtained. The three-modal concatenation vector is formed by concatenating the text semantic vector, visual feature vector, and coordinate feature vector in sequence (total dimension 1796) and using it as the input to the error correction network. The learningable weight matrix optimizes the modal weight mapping relationship through training. The three output nodes correspond to the weight parameters of text, vision, and coordinates, respectively. It serves as a bias vector, enhancing the model's ability to fit nonlinear weight relationships. The Sigmoid activation function compresses the output to the [0,1] interval, generating a modality weight vector. , respectively, represent the contribution of text, visual, and coordinate modalities in the fusion process, namely text weight, visual weight, and coordinate weight. To perform element-wise weighted operations, the modal features are dynamically weighted and summed using modal weights to generate a multimodal fusion feature F.

[0148] In an exemplary embodiment, a typical conflict resolution scenario is provided: Tesseract misidentifies "monetary funds" as "monetary assets" (text confidence score of 0.65), but YOLOX detects that the text is located in the table title area (visual confidence score of 0.95, coordinate confidence score of 0.98). The error correction mechanism outputs weights G=[0.2,0.6,0.2], with the visual modality weight significantly higher than the text modality. The multimodal fusion feature F, dominated by 60% visual features and combined with the semantic constraints of the table title, automatically corrects the text to "monetary funds," achieving cross-modal collaborative error correction.

[0149] In a specific application example, step 6 generates third structured data based on the multimodal fusion features, including steps 66 to 67.

[0150] Step 66: Perform element classification and coordinate metadata appending on the multimodal fusion features, and convert the multimodal fusion features into structured elements with semantic labels.

[0151] Attention scores are used to verify the semantic consistency of elements. For example, if the attention score between a text block and a table area is >0.8, the text is reinforced as a table title (marked as "TableTitle"), avoiding category errors caused by OCR misidentification. Furthermore, attention scores are combined with spatial proximity to construct parent-child relationships between elements. For example, the title "Balance Sheet" has an attention score of 0.91 with the table, and a vertical distance of 28 pixels < 50 pixels, establishing a hierarchical association "Title→Table," which is recorded in the JSON output via a parent field (e.g., "parent":"Title_001"). The physical order of elements is restored based on spatial coordinates, and a logical hierarchy is established through semantic relationships, ensuring that the structured result conforms to human reading habits.

[0152] The process of element classification is as follows.

[0153] (1) Text element classification: SpaCy is used to perform semantic boundary segmentation on the searchable text stream, combined with the font size (e.g., ≥18pt is marked as Title), spatial position (Y<20% of page height is top title) and regular expressions extracted by PDFMiner (e.g., ... Identify footnotes, generate Text elements, and tag them with semantic labels (Title / Body / Footnote). If the text is corrected (e.g., "Currency Funds" → "Cash and Cash Equivalents"), the corrected text content will be used, and the confidence score will be updated. Example: The cover title "2024 Q3 Financial Report" is tagged as a Title type because the font size is ≥18pt and it is located at the top of the page; the footnote "Data as of September 30, 2024" is tagged as a Title type because it contains "...". The symbol is located at the bottom of the page and is marked as Footnote.

[0154] The final output is a structured Text element, including semantic tags (Title / Body / Footnote), coordinate metadata (PDF point space and pixel space coordinates), and font style information. Example: The cover title "2024 Q3 Financial Report" is tagged as "Title," with coordinates recorded as (X1=100, Y1=272, X2=200, Y2=292), and marked with the spatial attribute "centered at the top of the page."

[0155] (2) Table Element Classification: Based on the table bounding boxes detected by the YOLOX model (e.g., x1=150, y1=300), TesseractOCR is used to identify the cell regions. Rows are segmented by vertical coordinate spacing (e.g., row spacing > 20 pixels), and columns are segmented by horizontal coordinate spacing combined with semantic tags (e.g., "subject", "amount") to infer the relationship between the table header and data rows. An HTML table with semantic tags is generated. Example: After the detected table regions are identified by OCR, a balance sheet structure is generated based on the row and column segmentation results and marked as Table type.

[0156] The final output is a structured Table element, containing HTML-formatted table data, semantic inference results for rows and columns (such as table headers and data rows), and coordinate metadata. Example: A table region (X1=150, Y1=300, X2=550, Y2=500) detected by the YOLOX model is processed by Tesseract to generate an HTML table with "Subject" and "Amount" columns, and the row structure is segmented using vertical spacing (30 pixels > 20 pixels threshold).

[0157] (3) Image element classification: Based on the classification results of the YOLOX model, an Image element is generated, and the pixel spatial coordinates (after rect_to_bbox calibration) are recorded. The image format (PNG / JPEG), coordinates, and category label are recorded, such as {"type":"Image","category":"Logo"}. Example: An embedded LOGO is detected by YOLOX as belonging to the "Logo" category, an Image element is generated, and the coordinates are recorded.

[0158] In a specific application example, coordinate standardization is first performed based on pixel space coordinates (origin at the top left corner, Y-axis increasing downwards) to unify the coordinate baseline of all elements. Two-dimensional sorting rules prioritize the X-axis, arranging elements in ascending order based on their left boundary coordinate (X1) to ensure left-to-right order (e.g., left column text in multi-column layouts are sorted first); the Y-axis is secondary, arranging elements in the same column in ascending order based on their top boundary coordinate (Y1) to ensure top-to-bottom order (e.g., title → table → footnote). When the difference in X1 values ​​between elements exceeds the column width threshold (default 50 pixels), they are considered different column areas, sorted separately, and then merged (e.g., left column text and right column footnotes are sorted independently). If the vertical distance between a child element and its parent element is less than 50 pixels and they belong to the same page, they are marked as parent-child elements. Example: The vertical distance between the table title (Y=272) and the table area (Y=300) is 28 pixels (less than 50 pixels), establishing a "title-table" parent-child relationship. Semantic-assisted verification, combined with element type tags (e.g., the logical relationship between Title and Table), improves the accuracy of hierarchy judgment.

[0159] Metadata appending refers to adding extra data describing the attributes, relationships, and characteristics of each element when parsing a PDF document into structured elements (such as Text, Table, Image, etc.). This data is not directly related to the element's content itself (such as the specific text in the text or the numerical values ​​in the table), but rather describes the element's role, position, and relationships within the document, and is stored as metadata fields in JSON format.

[0160] The process of attaching coordinate metadata is as follows.

[0161] (1) Coordinate system transformation and normalization: The rect_to_bbox function is used to transform the coordinate system and resolve the difference between the origin at the bottom left corner of the PDF and the origin at the top left corner of the image. The pixel coordinates are divided by the image width / height and scaled to the [0,1] range. Example: The point coordinates (100pt, 520pt) of the PDF are converted to pixel coordinates (416.67px, 216.67px). When the image width is 800px and the height is 600px, the normalized coordinates are (0.5208, 0.3611).

[0162] (2) Rules for attaching coordinate metadata: Text elements are attached with font size, hierarchical labels and pixel / point spatial coordinates, and table / image elements are attached with bounding box coordinates detected by YOLOX and normalized coordinates.

[0163] The following effects can be achieved by attaching metadata.

[0164] 1) Construct a logical hierarchy for the document, using the `parent` and `children` fields to clearly define the logical relationships between elements. For example, a table can be associated with its title as the parent element and its footnotes as child elements, forming a hierarchical structure of "Title→Table→Footnote," which facilitates logical navigation and understanding of the document. Example: `metadata:{"parent":"Title_001","children":["Footnote_003"]}` indicates that the table is associated with the title "Title_001" and has the footnote "Footnote_003" as a child element.

[0165] 2) Ensure elements are ordered. Use the `sort_order` field to record the order in which elements appear in the document, ensuring that the structured output conforms to human reading habits (such as left to right, top to bottom). For example, `sort_order:2` indicates that the element is the second main content block in the document.

[0166] 3) Supports result verification and traceability: Stores metadata such as the parsing confidence score and feature weights of elements, which facilitates subsequent verification of the reliability of the parsing results. For example, the metadata of a table element can record its visual detection confidence score (0.95) and the confidence score after text correction (0.812).

[0167] 4) Enhance the semantics of structured data: Add semantic categories (such as “BalanceSheet”), coordinate spaces (such as pixel_space), and format information (such as font size) to structured data to give it richer semantic information and make it suitable for downstream AI applications.

[0168] This application utilizes multimodal feature fusion to convert unstructured elements such as tables and images in PDFs into structured elements with semantic tags, solving the structural errors inherent in traditional single-modal parsing. For example, tables in scanned PDFs are transformed into computable table elements through YOLOX localization, OCR recognition, and semantic constraints, increasing accuracy from 65% to 95.1%. This structured representation of unstructured information converts unstructured text and images in PDFs into standardized elements that computers can understand, overcoming the limitations of traditional parsing which can only extract text streams or image pixels. Furthermore, by combining BERT semantic vectors with coordinate space relationships, it can understand the document's logical hierarchy (e.g., title → body text → footnotes), rather than simply performing pixel-level parsing. For example, the text "cash and cash equivalents" in a financial report is correctly classified as a table-related element through semantic-spatial dual association with the table's amount area. The generated structured table elements (e.g., a balance sheet) can directly extract data such as "beginning balance of cash and cash equivalents: 50 million yuan," reducing manual input errors and adapting to the high-frequency financial report parsing needs. By using hierarchical structure (such as clause paragraphs → appendix charts), key clauses in contracts can be quickly located, improving document review efficiency by over 30%. Scanned documents, after processing in step 66, generate Text / Table elements with coordinate metadata, supporting subsequent automated retrieval and structured management. Image and Text elements are spatially aligned using coordinate metadata; for example, the text "LOGO" is associated with its corresponding Image element on the coordinate plane, forming a "text-image" semantic pair.

[0169] Step 67: Populate the JSON template with the structured elements to obtain the third structured data. For example, the HTML structure of the Table element is converted into an "html" field. The element's "metadata" fields (such as coordinates, weights) are mapped to the "metadata" block.

[0170] In a specific application example, the decision-making process is first visualized: attention scores are stored as metadata in the `metadata.attention_scores` field of the JSON, recording intermediate results of multimodal fusion for subsequent verification and error correction. Then, the output is optimized: attention scores guide element sorting and layout restoration. For example, elements are sorted in descending order of attention scores to ensure that key content (such as highly relevant title-table pairs) is output first, conforming to human reading habits. When tables are included, `Table` elements with HTML structure are output. For plain text, `Text` elements and semantic tags are output.

[0171] The predefined structure of the JSON template is shown in Table 4.

[0172] Table 4 Core Field Template Description

[0173]

[0174] The data mapping rules for concatenating multimodal data into JSON are shown in Table 5.

[0175] Table 5 Data Mapping Rules

[0176]

[0177] Compared with the prior art, this application has the following beneficial effects.

[0178] (1) Improved table recognition accuracy: Table regions are located using OpenCV edge detection + YOLOX object detection, ensuring accurate recognition of high-confidence tables. YOLOX outputs the bounding box coordinates and category labels of the table, providing precise visual localization. A trimodal cross-attention mechanism fuses textual semantics (BERT vectors), visual features (YOLOX detection results), and coordinate features, strengthening the semantic-spatial association of table regions. A dynamic weighted error correction mechanism prioritizes high-confidence visual modalities (e.g., table detection confidence of 0.95) to correct OCR recognition errors and improve the accuracy of table content extraction.

[0179] (2) Reduced parsing efficiency for complex documents: It can quickly determine whether a document is searchable text. If pdf_text_extractable=True, it skips OCR and directly enters the fast_text_extract strategy. It automatically switches strategies based on layout complexity (such as whether it contains tables) to avoid redundant layout analysis for simple documents. It focuses on highly relevant areas through attention scores and performs fine-grained parsing only on key content (such as table titles), reducing the computational load of OCR.

[0180] (3) Improved document type coverage: Through a two-layer decision-making process of the auto strategy (text extractability → layout complexity), documents are divided into four scenarios (text documents without tables, text documents with tables, scanned documents without tables, and scanned documents with tables). The optimal strategy is matched for each scenario to avoid the limitations of a single strategy. Regardless of whether the document has extractable text, it can be parsed through text semantics (if any) or visual features (scanned documents). For example, scanned documents are processed by YOLOX+OCR, and text documents are processed by PDFMiner+semantic analysis to ensure the parsing feasibility of various types of documents.

[0181] (4) Enhanced traceability: By attaching decision-making process data to the structured output, each step of the analysis result is traceable. For example, the confidence score of 0.812 for the OCR error correction result "monetary funds" is calculated by visual weight 0.6 × 0.92 + text weight 0.4 × 0.65, which users can use to verify the reasonableness of the result. The relationship between text semantics, visual positioning, and coordinate space is explicitly recorded, which facilitates the verification of the consistency of multimodal information during manual review (such as whether the spatial proximity of the text content matches that of the table area).

[0182] Based on the same inventive concept, this application also provides a PDF document intelligent parsing device for implementing the methods described above. The solution provided by this device is similar to the implementation scheme described in the methods above; therefore, the specific limitations of one or more PDF document intelligent parsing device embodiments provided below can be found in the limitations of the methods described above, and will not be repeated here.

[0183] In one exemplary embodiment, such as Figure 3 As shown, a PDF document intelligent parsing device is provided, including: a preprocessing module 301, a text judgment module 302, a first parsing module 303, a table judgment module 304, a second parsing module 305, and a third parsing module 306.

[0184] The preprocessing module 301 is used to acquire the PDF document to be parsed, parse the page elements in the PDF document, and generate a document metadata dictionary. The document metadata dictionary includes the number of pages and the distribution of elements.

[0185] The text determination module 302 is used to determine whether the PDF document contains extractable text based on the document metadata dictionary.

[0186] The first parsing module 303 is used to convert the PDF document into an image and perform optical character recognition to generate first structured data when the PDF document does not contain extractable text.

[0187] The table determination module 304 is used to determine whether the PDF document contains a table when the PDF document contains extractable text, using traditional visual methods and deep learning algorithms.

[0188] The second parsing module 305 is used to extract text using PDFMiner and generate second structured data when the PDF document does not contain tables.

[0189] The third parsing module 306 is used to extract and fuse multimodal features from the PDF document according to the document metadata dictionary when the PDF document contains tables, to obtain multimodal fused features, and to generate third structured data based on the multimodal fused features. The first structured data, the second structured data, or the third structured data are the parsing results of the PDF document.

[0190] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0191] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0192] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0193] In this application, all actions to acquire signals, information, or data are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with the authorization granted by the owner of the relevant device.

[0194] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0195] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for intelligent parsing of unstructured PDF documents, characterized in that, The method includes: Obtain the PDF document to be parsed, parse the page elements in the PDF document, and generate a document metadata dictionary; the document metadata dictionary includes the number of pages and the distribution of elements; Based on the document metadata dictionary, determine whether the PDF document contains extractable text; If the PDF document does not contain extractable text, the PDF document is converted into an image and optical character recognition is performed to generate the first structured data. If the PDF document contains extractable text, traditional visual methods and deep learning algorithms are used to determine whether the PDF document contains a table. Specifically, this includes: using OpenCV edge detection algorithm and Hough transform to filter horizontal lines in the PDF document and calculating the horizontal line density; using deep learning algorithms to perform object detection on the unstructured PDF document to determine the table detection result; the table detection result includes table bounding box, confidence score, and category label; and determining whether the PDF document contains a table based on the horizontal line density and the table detection result. If the PDF document does not contain tables, PDFMiner is used to extract the text and generate the second structured data. If the PDF document contains tables, then based on the document metadata dictionary, multimodal feature extraction and feature fusion are performed on the PDF document to obtain multimodal fusion features, and third structured data is generated based on the multimodal fusion features; wherein, the first structured data, the second structured data, or the third structured data are the parsing results of the PDF document; Specifically, based on the document metadata dictionary, multimodal feature extraction and feature fusion are performed on the PDF document to obtain multimodal fused features, including: Based on the document metadata dictionary, the PDF document is subjected to text semantic tagging, image block calibration and coordinate calibration to obtain preprocessed text semantic data, preprocessed image data and preprocessed coordinate data. Feature extraction is performed on the preprocessed text semantic data, the preprocessed image data, and the preprocessed coordinate data respectively to obtain text semantic vector, visual feature vector, and coordinate feature vector; A trimodal cross-attention mechanism is adopted to perform cross-modal attention fusion on the text semantic vector, the visual feature vector and the coordinate feature vector to quantify the semantic-spatial correlation between the text and the visual elements; Based on the semantic-spatial correlation, the text semantic vector, the visual feature vector, and the coordinate feature vector, a pre-trained neural network model is used to dynamically adjust the weights of text, visual, and coordinate to obtain text weight, visual weight, and coordinate weight. Based on the text weight, the visual weight, and the coordinate weight, the text semantic vector, the visual feature vector, and the coordinate feature vector are weighted and summed to obtain the multimodal fusion feature.

2. The intelligent parsing method for unstructured PDF documents according to claim 1, characterized in that, Parsing the page elements in the PDF document to generate a document metadata dictionary specifically includes: The PDF document is parsed page by page using the pdfminer.high_level.extract_pages() function in Python to determine the number of pages and the distribution of elements in the PDF document, in order to generate a document metadata dictionary.

3. The intelligent parsing method for unstructured PDF documents according to claim 1, characterized in that, Based on the document metadata dictionary, it is determined whether the PDF document contains extractable text, specifically including: Based on the document metadata dictionary, the pdfminer.high_level.extract_text() function in Python is called to extract the text stream. If the text stream is successfully extracted, it is determined that the PDF document contains extractable text; otherwise, it is determined that the PDF document does not contain extractable text.

4. The intelligent parsing method for unstructured PDF documents according to claim 1, characterized in that, Feature extraction is performed on the preprocessed text semantic data, the preprocessed image data, and the preprocessed coordinate data to obtain text semantic vectors, visual feature vectors, and coordinate feature vectors, specifically including: The BERT model is used to extract the text semantic vectors from the preprocessed text semantic data; The YOLOX model is used to extract visual features from the preprocessed image data to obtain visual feature vectors. The preprocessed coordinate data is normalized to obtain coordinate feature vectors.

5. The intelligent parsing method for unstructured PDF documents according to claim 1, characterized in that, The generation of third-level structured data based on the multimodal fusion features specifically includes: The multimodal fusion features are classified into elements and have coordinate metadata appended, and then converted into structured elements with semantic labels. The structured elements are used to populate the JSON template to obtain the third structured data.

6. A device for intelligent parsing of unstructured PDF documents, characterized in that, The apparatus is applied to the intelligent parsing method for unstructured PDF documents according to any one of claims 1-5, and the apparatus comprises: The preprocessing module is used to obtain the PDF document to be parsed, parse the page elements in the PDF document, and generate a document metadata dictionary; the document metadata dictionary includes the number of pages and the distribution of elements; The text determination module is used to determine whether the PDF document contains extractable text based on the document metadata dictionary; The first parsing module is used to convert the PDF document into an image and perform optical character recognition to generate first structured data when the PDF document does not contain extractable text. The table determination module is used to determine whether the PDF document contains tables when the PDF document contains extractable text, using traditional visual methods and deep learning algorithms. The second parsing module is used to extract text using PDFMiner and generate second structured data when the PDF document does not contain tables. The third parsing module is used to extract and fuse multimodal features of the PDF document according to the document metadata dictionary when the PDF document contains tables, to obtain multimodal fused features, and to generate third structured data according to the multimodal fused features; wherein the first structured data, the second structured data, or the third structured data are the parsing results of the PDF document.

7. A computer device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the intelligent parsing method for unstructured PDF documents according to any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the intelligent parsing method for unstructured PDF documents as described in any one of claims 1-5.

Images