Intelligent analysis method and device for unstructured PDF document, equipment and medium

Abstract

The invention discloses an intelligent analysis method and device for an unstructured PDF document, equipment and a medium, and relates to the field of document analysis, the method comprises the steps of obtaining a to-be-analyzed PDF document, analyzing page elements in the PDF document, and generating a document metadata dictionary; if the PDF document does not contain the extractable text, converting the PDF document into an image and performing optical character recognition to generate first structured data; if the PDF document contains the extractable text, judging whether the PDF document contains a table or not; if the PDF document does not contain the table, a PDFMiner is adopted to extract the text, and second structured data is generated; if the PDF document contains the table, performing multi-modal feature extraction and feature fusion on the PDF document according to the document metadata dictionary to obtain multi-modal fusion features, and generating third structured data according to the multi-modal fusion features; according to the method and the device, the analysis precision and efficiency of the PDF document are improved.

Description

Technical Field

[0001] The present application relates to the field of document parsing, and in particular to a method, apparatus, device and medium for intelligent parsing of unstructured PDF documents. Background Art

[0002] Currently, the common parsing methods for Portable Document Format (PDF) documents mainly include: (1) Text extraction technology: pure text parsing based on PDFMiner, which is suitable for PDFs where text can be directly extracted, but is powerless for scanned or imaged text; (2) Optical Character Recognition (OCR) technology: optical character recognition based on Tesseract, which is suitable for scanned documents, but has shortcomings in text block segmentation and semantic understanding of complex layouts; (3) Layout analysis technology: document layout detection based on deep learning models (such as YOLOX), which can recognize areas such as text boxes, images, and tables, but lacks a multimodal decision fusion mechanism.

[0003] In addition to the above problems, existing technologies also have the following defects: (1) Limitations of single-modal processing: Traditional PDF parsing methods rely only on text extraction or single OCR technology. When faced with complex layouts (such as charts, formulas, and images embedded in text), information omissions or structural disorder often occur; (2) Single strategy: Existing systems lack a dynamic decision-making mechanism and cannot automatically select the optimal parsing strategy based on document characteristics (such as text extractability and image complexity), resulting in limited parsing efficiency and accuracy; (3) Insufficient multimodal fusion: The system fails to achieve deep fusion of multimodal information such as text semantics, image layout, and coordinate positioning, making it difficult to accurately understand the logical structure of the document (such as tables, lists, and paragraph levels). Summary of the Invention

[0004] The purpose of this application is to provide a method, device, equipment and medium for intelligent parsing of unstructured PDF documents, which can improve the parsing accuracy and efficiency of PDF documents.

[0005] To achieve the above objectives, this application provides the following solutions: In a first aspect, the present application provides a method for intelligent parsing of unstructured PDF documents, comprising: Obtain a PDF document to be parsed, parse the page elements in the PDF document, and generate a document metadata dictionary; the document metadata dictionary includes the number of pages and element distribution; Determining whether the PDF document contains extractable text based on the document metadata dictionary; If the PDF document does not contain extractable text, converting the PDF document into an image and performing optical character recognition to generate first structured data; If the PDF document contains extractable text, traditional visual methods and deep learning algorithms are used to determine whether the PDF document contains a table; If the PDF document does not contain a table, extracting text using PDFMiner to generate second structured data; If the PDF document contains a table, multimodal feature extraction and feature fusion are performed on the PDF document according to the document metadata dictionary to obtain multimodal fusion features, and third structured data is generated based on the multimodal fusion features; wherein the first structured data, the second structured data, or the third structured data is the parsing result of the PDF document.

[0006] In a second aspect, the present application provides an unstructured PDF document intelligent parsing device, comprising: A preprocessing module is used to obtain a PDF document to be parsed, parse the page elements in the PDF document, and generate a document metadata dictionary; the document metadata dictionary includes the number of pages and element distribution; A text determination module, configured to determine whether the PDF document contains extractable text based on the document metadata dictionary; A first parsing module is configured to convert the PDF document into an image and perform optical character recognition to generate first structured data when the PDF document does not contain extractable text; A table determination module is used to determine whether the PDF document contains a table by using traditional visual methods and deep learning algorithms when the PDF document contains extractable text; A second parsing module is configured to extract text using PDFMiner to generate second structured data when the PDF document does not contain a table; a third parsing module configured to, when the PDF document contains a table, perform multimodal feature extraction and feature fusion on the PDF document according to the document metadata dictionary to obtain multimodal fusion features, and generate third structured data based on the multimodal fusion features; wherein the first structured data, the second structured data, or the third structured data is a parsing result of the PDF document.

[0007] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned method for intelligent parsing of unstructured PDF documents.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned unstructured PDF document intelligent parsing method.

[0009] According to the specific embodiments provided in this application, this application has the following technical effects: The present application provides a method, apparatus, device and medium for intelligent parsing of unstructured PDF documents. By determining whether a PDF document contains extractable text and whether it contains tables, different parsing strategies are selected, and multimodal data such as text and images are integrated. In combination with a dynamic strategy selection mechanism, high-precision parsing and structured output of unstructured documents are achieved, thereby solving the problems of single-modal processing, rigid strategies and insufficient multimodal fusion in the existing technology, and improving the parsing accuracy and efficiency of PDF documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A schematic diagram of the overall process of an unstructured PDF document intelligent parsing method provided in one embodiment of the present application.

[0012] Figure 2 A detailed flowchart of an unstructured PDF document intelligent parsing method provided in one embodiment of the present application.

[0013] Figure 3 A schematic diagram of the functional modules of an unstructured PDF document intelligent parsing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0015] The purpose of this application is to fuse multimodal data such as text and images, and combine it with a dynamic strategy selection mechanism to achieve high-precision parsing and structured output of unstructured documents, thereby solving the problems of single-modal processing, strategy rigidity and insufficient multimodal fusion in the existing technology, and automatically select the best parsing strategy based on the document type (text extractable / scanned) and complexity (whether it contains tables).

[0016] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0017] First, some technical terms involved in the embodiments of this application are introduced.

[0018] 1. PDFMiner is a Python library that focuses on accurately extracting text, images, tables and other data from PDF files. It is particularly good at processing complex format documents such as academic papers, contracts, reports, etc., and can accurately restore content structure and semantic information.

[0019] pdf2image is a Python library that converts PDF files to image formats like PNG and JPEG. It's often used to extract images from PDFs, process scanned documents, or serve as an OCR preprocessor. It leverages underlying PDF rendering tools like Poppler to achieve this conversion, rendering each PDF page as a high-resolution image and providing an easy-to-use API.

[0020] 3. YOLOX is an advanced version of the YOLO series, developed by the Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences in 2021. While maintaining YOLO's high-speed detection advantage, it introduces innovative technologies to significantly improve detection accuracy, especially for small objects and densely populated scenes. It is an open-source object detection algorithm.

[0021] 4. The Bidirectional Encoder Representations from Transformers (BERT) model is a pre-trained language model developed by Google in 2018. Based on the Transformer architecture, it significantly improves the performance of natural language processing (NLP) tasks through bidirectional contextual learning technology, ushering in a new era of "pre-training + fine-tuning" in NLP and has become a foundational model for various NLP applications.

[0022] 5. spaCy is an industrial-grade NLP library developed in Python and Cython that efficiently processes and analyzes large amounts of text data. It provides pre-trained models, multi-language support, and a concise application programming interface (API). It quickly implements common NLP tasks such as word segmentation, part-of-speech tagging, named entity recognition (NER), and dependency parsing. It is widely used in fields such as information extraction, text classification, and semantic understanding.

[0023] 6. OCR is a technology that converts text in images into machine-readable text. With the help of computer vision and machine learning algorithms, it recognizes text in images (such as printed text and handwritten text) and converts it into text format. It is widely used in document digitization, information extraction, image translation and other scenarios.

[0024] 7. Tesseract OCR is an open source OCR engine that provides a Python interface and can convert text in images into machine-readable text format. It supports over 100 languages, including Chinese, English, Japanese, Korean, etc., and has a wide range of applications in document scanning, image translation, license plate recognition, digital libraries, and other fields.

[0025] In an exemplary embodiment, Figure 1 and Figure 2 As shown, a method for intelligent parsing of a PDF document is provided, which is executed by a computer device and includes steps 1 to 6.

[0026] This application first designs four strategy modes.

[0027] (1) auto: Automatic strategy, defines the identification variable pdf_text_extractable according to the success or failure of text extraction, and defines the identification variable infer_table_structure according to the success or failure of table recognition. Based on the positive or negative direction of the two variables, the parsing strategy is dynamically selected.

[0028] (2) ocr_only: Pure OCR extraction strategy, execute step 3. Convert the PDF page to an image (e.g., 300 DPI) using pdf2image, without performing block segmentation or layout analysis. Call Tesseract to perform OCR on the entire page image, generating a text stream. Element classification: Only text elements are generated, without performing table or hierarchical analysis (due to the lack of layout information). Finally, a structured output containing only text elements is generated.

[0029] (3) fast_text_extract: Fast text extraction strategy, execute step 5. Skip complex steps such as image segmentation and YOLOX detection and directly extract searchable text through PDFMiner. Text cleaning: Remove redundant spaces and line breaks, and standardize the format through regular expressions (e.g., re.sub(r'[\n\t\r]+','')). Hierarchy marking: Based on font size (e.g., ≥18pt for Title) and spaCy paragraph segmentation, construct a simple hierarchy (Title→Body), and finally generate a lightweight structured output that focuses on text content and simple hierarchy.

[0030] (4) layout_resolve: High-precision layout analysis + OCR strategy, execute step 6, infer the table row and column structure based on the fusion features, generate an HTML table with semantic tags, and finally generate a JSON / XML output with a complete table structure.

[0031] In summary, the detailed principles of the four strategies are shown in Table 1.

[0032] Table 1 Four strategies

[0033] The four strategies cover more than 95% of document types through a dynamic decision-making mechanism, achieving "quick parsing of simple documents and accurate parsing of complex documents", and ultimately uniformly output standardized structured data to support intelligent applications in finance, law and other fields.

[0034] In actual application, you can specify a strategy for PDF parsing, or you can follow the default process and dynamically select the parsing strategy to adapt to different application scenarios.

[0035] Step 1: Obtain a PDF document to be parsed, parse the page elements in the PDF document, and generate a document metadata dictionary. The document metadata dictionary includes the number of pages and element distribution.

[0036] In a specific application example, the pdfminer.high_level.extract_pages() function in Python is called to parse the PDF document page by page, determine the page count and element distribution of the PDF document, and generate a document metadata dictionary. Specifically, a user-uploaded PDF document (assuming it is (2024Q3 Financial Report.pdf)) is loaded through the file input interface (pdfminer library). pdfminer.high_level.extract_pages() is called to parse the PDF document page by page, obtain the element types and coordinates of text boxes, images, tables, and other elements, and generate a document metadata dictionary containing the page count and element distribution.

[0037] Step 2: Based on the document metadata dictionary, determine whether the PDF document contains extractable text.

[0038] In a specific application example, based on the document metadata dictionary, the pdfminer.high_level.extract_text() function in Python is called to extract the text stream. If the text stream is extracted successfully, the PDF document contains extractable text; otherwise, the PDF document does not contain extractable text.

[0039] Step 3: If the PDF document does not contain extractable text, convert the PDF document into an image and perform optical character recognition to generate first structured data.

[0040] Specifically, if the pdfminer.high_level.extract_text() function returns empty text, that is, the PDF document does not contain extractable text, then assign pdf_text_extractable=False, directly triggering the ocr_only strategy for parsing.

[0041] Step 4: If the PDF document contains extractable text, traditional visual methods and deep learning algorithms are used to determine whether the PDF document contains a table.

[0042] Furthermore, if the PDF document contains extractable text (i.e., content such as cover title, body paragraphs, etc.), pdf_text_extractable=True is assigned, and the text cleaning and preprocessing process is performed, such as steps 41 to 43 below.

[0043] Step 41: Format standardization. Use the regular expression re.sub(r'[\n\t\r]+', ' ') to remove spaces from the text. For example, convert "Cash and cash equivalents\n\t\t50 million yuan" to "Cash and cash equivalents 50 million yuan" to improve the readability of the text.

[0044] In regular expressions, re.sub is a function in Python that replaces the matching content in a string. r'[\n\t\r]+' is a regular expression used to match specific whitespace characters. "r" represents the original string, avoiding escaping the backslash. "[...]" is a character set, which means matching any character in the square brackets. " is a newline character, "\t" is a tab character, "\r" is a carriage return character, and "+" indicates that the preceding character or character set can appear one or more times. "' '" (space) is the replacement content of the re.sub function, indicating that the matched whitespace character will be replaced with a single space. The above regular expression can effectively replace multiple whitespace characters in a string with a single space, thereby cleaning up the redundant whitespace in the text.

[0045] Step 42, paragraph segmentation, uses spaCy's en_core_web_sm model to segment the text paragraphs according to semantic boundaries (for example, "Important Accounting Policy Statement" is divided into three natural paragraphs, "Important", "Accounting", and "Policy Statement") to build a logical hierarchical foundation.

[0046] Among them, en_core_web_sm is a small English language model provided by spaCy, specifically for NLP tasks. It is a pre-trained model based on deep learning technology and can handle a variety of common NLP tasks, such as word segmentation, part-of-speech tagging, dependency parsing, and named entity recognition (NER).

[0047] Step 43, special mark processing, through regular expressions Identify footnotes (such as " ”, extract “Data as of September 30, 2024”, and combine the coordinates extracted by PDFMiner (such as the Y axis is greater than 80% of the page height) to mark “Data as of September 30, 2024” as a footer type.

[0048] This application can achieve the following three beneficial effects through the text cleaning and preprocessing steps from step 41 to step 43.

[0049] (1) Improve the accuracy of parsing: Format standardization eliminates interference from text format, making subsequent understanding and processing of text content more accurate; paragraph segmentation rationally divides text into logical units, and special marks mark key information, which helps to fully and accurately understand the document content, reduce parsing errors caused by improper text processing, and improve the overall parsing quality.

[0050] (2) Text cleaning and preprocessing can lay the foundation for subsequent multimodal processing, feature fusion, structured output and other steps, ensuring the smooth progress of the entire parsing process. If these preprocessing steps are missing, subsequent text-based analysis and processing will face problems such as data confusion and semantic ambiguity, affecting the normal operation of the parsing system and the final results.

[0051] (3) Enhanced parsing adaptability: Through standardized processing, paragraph segmentation and special marking of texts of various formats and types, it can adapt to a variety of PDF documents, whether they are academic papers, financial reports or legal contracts, and can effectively process the text content therein, thus expanding the scope of application and applicability.

[0052] In a specific application example, step 4 uses traditional visual methods and deep learning algorithms to determine whether the PDF document contains a table, which specifically includes the following steps 44 to 46.

[0053] Step 44 , using the OpenCV edge detection algorithm and Hough transform to screen the horizontal lines in the PDF document, and calculate the density of the horizontal lines (e.g., 8 horizontal lines are detected in a 100×100 pixel grid).

[0054] Step 45: Perform object detection on the PDF document using a deep learning algorithm to determine a table detection result. The table detection result includes a table bounding box, a confidence score, and a category label. The deep learning algorithm is a YOLOX model.

[0055] This application uses a lightweight model (such as yolox-nano.onnx) to detect PDF documents and output table bounding boxes (such as x1=150, y1=300, x2=550, y2=500), confidence (0.92), and category labels. Category labels include Title, Table, Footnote, etc.

[0056] Step 46: Determine whether the PDF document contains a table based on the horizontal line density and the table detection result.

[0057] Specifically, if the horizontal line density is ≥ 4 lines / 100 pixels squared and the confidence level detected by the YOLOX model is ≥ 0.8, it is determined that a high-confidence table exists, infer_table_structure=True is assigned, and the layout_resolve strategy is enabled.

[0058] Step 5: If the PDF document does not contain a table, PDFMiner is used to extract text (ie, the fast_text_extract strategy is executed) to generate second structured data.

[0059] In step 6, if the PDF document contains a table, multimodal feature extraction and feature fusion are performed on the PDF document based on the document metadata dictionary to obtain multimodal fusion features, and third structured data is generated based on the multimodal fusion features. The first structured data, the second structured data, or the third structured data is the result of parsing the PDF document.

[0060] In a specific application example, in step 6, multimodal feature extraction and feature fusion are performed on the PDF document according to the document metadata dictionary to obtain multimodal fusion features, including the following steps 61 to 65.

[0061] Step 61 : performing text semantic tagging, image block calibration, and coordinate calibration processing on the PDF document according to the document metadata dictionary to obtain pre-processed text semantic data, pre-processed image data, and pre-processed coordinate data.

[0062] (1) Text Semantic Marking: Entities in the preprocessed text semantic data were extracted using spaCy (e.g., “currency funds” was marked as FINANCIAL_ENTITY), and font sizes were extracted using the LTChar.size property of PDFMiner (e.g., ≥18pt was marked as Title, and 12pt was marked as Body). A hierarchical relationship between Title and Body was constructed. This hierarchical relationship was constructed based on the triple constraints of font size, spatial position, and semantic association, forming a logical chain from title to body. Specifically, font size was used as the leading indicator, as shown in Table 2, and spatial position was used to assist in verification.

[0063] Table 2 Font size

[0064] The title positioning rule is as follows: in the PDF document's point space, the title's Y-axis coordinate must be less than 20% of the page height (adjustable based on document type, such as 15% for academic papers and 20% for reports). The title's X-axis coordinate must be centered within the page width, with a deviation of no more than 10% of the page width. Through the multimodal collaboration of spatial location, semantic features, and visual positioning, the title positioning rule becomes the key link between the document's physical layout and logical semantics. Its accuracy directly impacts the quality of all subsequent parsing steps and is one of the core technologies fundamental to intelligent parsing of unstructured PDFs.

[0065] In a specific example, the conversion process from pixel to point space is: given DPI = 300, 1 point (pt) = Pixels, so the formula for converting pixel coordinates to point coordinates is: ; Where DPI is pixels per inch.

[0066] The calculation process of page height is: If the page height is 600 pixels, the formula for converting to point space is: .

[0067] The conversion process of the Y-axis coordinate is: if the Y-axis pixel coordinate of the Title is 120 pixels, the formula for converting it to point coordinates is: .

[0068] The threshold judgment process is: 20% of the page height is At this time, the Y-axis coordinate of Title is 28.8 points, which meets the Title positioning conditions.

[0069] Title positioning serves as the starting point for constructing a document's logical hierarchy. Title positioning determines the root node of each chapter, and combined with font size (e.g., ≥18pt) and semantic tags (e.g., the "balance sheet" keyword extracted by BERT), the "Title→Body" hierarchy is constructed. For example, after the title "2024 Q3 Financial Report" is positioned, paragraphs within a vertical distance of <50 pixels below it are automatically classified as the body text, forming a logical chain. Further verification of the title's visual positioning (the title area detected by YOLOX), text semantics (BERT vector), and coordinate space (the top area on the Y-axis) ensures the accuracy of the hierarchical relationship.

[0070] Furthermore, if the vertical distance between the Title and the subsequent table is less than 50 pixels, a "Title→Table" association is established through the parent field. The attention score of the Title area is usually >0.9, which can guide OCR to prioritize parsing title-related content and improve the efficiency of key information extraction. If the Title text is mistakenly identified by OCR as "2024Q3 Financial Report", the cosine similarity between the BERT vector and "Financial Report" is >0.85. Combined with the Title positioning rule (top of the Y axis + center of the X axis), dynamic weight correction is triggered (dominated by visual weight 0.6), and the text is corrected to the correct text. The unified Title positioning rule can ensure the standardization of the hierarchical structure of different documents, which is convenient for batch processing and retrieval (such as quickly locating financial report chapters through title keywords).

[0071] This application uses the logical hierarchy constructed by Title positioning to increase the accuracy of document element classification to over 95%, avoiding the hierarchical confusion problem of traditional single-modal parsing. Title serves as the root node of the hierarchy and supports the hierarchy field of JSON / XML output (such as {"id":"Title_001","children":["Table_001"]}), which is adapted to the intelligent retrieval needs of finance, law and other fields. In financial report analysis, it can quickly locate the "Balance Sheet" title, associate subsequent table data, and automatically extract key values ​​such as "Cash and Cash", reducing manual entry errors. In legal contract review, the "Breach of Contract Liability" section is located through Title, and the attachments and charts are associated, which improves the efficiency of clause retrieval by over 30%. In digital archive management, the Title of the scanned archive is OCR-recognized and located, and a searchable electronic catalog is established to achieve automatic classification of archives.

[0072] The positioning rule for the body section is: immediately below the title, with a vertical distance of less than 50 pixels and a font size of 12pt. SpaCy entity recognition results verify the logical relationship: if the title contains a FINANCIAL_ENTITY entity (such as "Balance Sheet"), the body section below it must contain similar entities (such as "Cash Funds" and "Opening Balance"). Otherwise, a hierarchy conflict check is triggered. For example, if the paragraph below the title "Balance Sheet" (FINANCIAL_ENTITY) contains "Cash Funds" (FINANCIAL_ENTITY), the Title→Body association is strengthened. If the "Company Logo" (a non-financial entity) appears, it is considered an independent image element and is not included in the Body hierarchy.

[0073] (2) Image Block Calibration: First, the pdf2image function is used to convert the page containing the table into a 300DPI image (800×600 pixels). Then, a sliding window is used to segment the image into 10×10 pixel sub-blocks (4800 blocks in total). Perspective transformation (cv2.findHomography) is then applied to correct and tilt the image (the error is reduced from 15° to <1°).

[0074] In PDF document parsing, chunking refers to the process of dividing a PDF page into multiple small pixel sub-blocks in both the horizontal and vertical directions after converting it to an image. Horizontal chunking refers to chunking along the width of the image, and the resulting number of sub-blocks is called the horizontal chunking number. Vertical chunking refers to chunking along the height of the image, and the resulting number of sub-blocks is called the vertical chunking number. The total number of chunks is the product of the horizontal and vertical chunking numbers.

[0075] For example, if an image is 800 pixels wide and 600 pixels high and is divided into 10×10 pixel sub-blocks, the number of horizontal blocks is: 800 pixels wide, each sub-block is 10 pixels wide, so the image can be divided into 800 ÷ 10 = 80 blocks. The number of vertical blocks is: 600 pixels high, each sub-block is 10 pixels high, so the image can be divided into 600 ÷ 10 = 60 blocks. The total number of blocks is: 80 horizontal blocks × 60 vertical blocks = 4800 small image blocks.

[0076] By dividing large images into small sub-blocks, it can adapt to the input requirements of lightweight object detection models (such as YOLOX-nano), reduce the computational complexity of a single inference, avoid small elements (such as footnotes and icons) being ignored in large images, and improve the ability to capture details through sub-block level detection.

[0077] After block division, a single YOLOX inference process only processes 10×10 pixel sub-blocks, which reduces the computational complexity compared to full-image detection (800×600 pixels). The parsing speed has been increased from 12.7s / page to 8.3s / page. Small sub-blocks prevent objects from being compressed or obscured in larger images. For example, a 10×10 pixel sub-block can accurately capture a 5×5 pixel footnote icon, which is easily missed by traditional full-image detection.

[0078] Documents containing multiple columns of text and nested tables can be parsed block by block after segmentation, avoiding layout confusion when parsing the entire image. For example, the terms and annexes of a legal contract are segmented, and the visual features of each sub-block are independently detected, and then the overall structure is restored through coordinate merging. Scanned PDFs are segmented and OCRed independently for each sub-block, reducing text distortion errors in the overall image OCR and improving the parsing accuracy of scanned documents to 87.5%.

[0079] Block segmentation is a key operation in PDF parsing that "breaks the whole into parts". By reducing computational complexity, improving detection accuracy, and supporting spatial alignment, it becomes a bridge connecting image preprocessing and multimodal feature extraction, ultimately ensuring efficient and accurate parsing of complex documents. This is especially indispensable in scenarios with high layout requirements, such as financial reports and legal contracts.

[0080] In one example, the coordinates of the upper-left corner and lower-right corner of the sub-block in the i-th row (vertically, starting at 0) and the j-th column (horizontally, starting at 0) are: (x1, y1) = (j×10, i×10), (x2, y2) = ((j+1)×10, (i+1)×10). For example, the coordinates of the sub-block in the 1st row and 1st column are (10, 10, 20, 20).

[0081] (3) Coordinate calibration: Use the rect_to_bbox function to convert the coordinates of the PDF point space to the coordinates of the pixel space. The default coordinates of the PDF are based on the lower left corner of the page as the origin. When moving to the right (X axis) and upward (Y axis), the coordinate values ​​gradually increase. However, in image processing, the coordinates are based on the upper left corner of the page as the origin. When moving to the right (X axis) and downward (Y axis), the coordinate values ​​gradually increase, which is completely opposite to the Y axis direction of the PDF. Therefore, the coordinates of the PDF points need to be converted to pixel coordinates.

[0082] First, convert the PDF's point coordinates (such as the location of text or images) to pixel coordinates, taking into account the document's resolution (DPI). For example, at 300 DPI, 1 point in PDF is approximately equal to 1.25 pixels (because 72 points = 1 inch and 300 pixels = 1 inch, so 1 point ≈ 300 / 72 ≈ 4.167 pixels). The conversion logic is: the larger the PDF's X coordinate value, the larger the corresponding X value of the pixel coordinate, and the direction is consistent. For example, if X = 100 points in PDF, simply multiply by the pixel / point ratio when converting to pixels. PDF's Y coordinate originates from the lower-left corner, while pixel coordinates originate from the upper-left corner, so the Y axis needs to be "flipped"—subtracting the pixel value corresponding to the PDF's Y coordinate from the total pixel height of the page to obtain the Y coordinate in pixel space.

[0083] Then, the orientation and origin are corrected: the PDF's Y axis grows upward, while the pixel space's Y axis grows downward. Mathematical operations are performed to "flip" the PDF's Y coordinates to the correct orientation in pixel space. For example, if the Y coordinate of an element in the PDF is 500 points and the total page height is 842 points, when converting to pixels, the number of pixels corresponding to this value must be subtracted from the total pixel height to ensure that the element's position in the image matches the PDF.

[0084] The rect_to_bbox function eliminates the differences in the coordinate systems between PDFs and images and unifies the positions of all elements into pixel space. This allows text semantics, visual layout, and coordinate positions to be associated on the same basis, providing an accurate spatial positioning foundation for subsequent multimodal detection, error correction, and structured parsing.

[0085] After the block coordinates and the text point coordinates extracted by PDFMiner are converted through rect_to_bbox, they can accurately match the positions of text and visual elements (for example, the text "Balance Sheet" corresponds to the table area within the sub-block), providing a spatial reference for the attention score calculation in the subsequent step 63. Furthermore, the adjacent relationship between sub-blocks (such as upper and lower, left and right sub-blocks) can help infer the logical hierarchy of elements. For example, a title and a table in vertically adjacent sub-blocks are determined to be related content.

[0086] Step 62 : performing feature extraction on the pre-processed text semantic data, the pre-processed image data and the pre-processed coordinate data respectively to obtain a text semantic vector, a visual feature vector and a coordinate feature vector.

[0087] In a specific application example, a BERT model is used to extract text semantic vectors from the preprocessed text semantic data. A YOLOX model is used to extract visual features from the preprocessed image data to obtain visual feature vectors. The preprocessed coordinate data is normalized to obtain coordinate feature vectors.

[0088] (1) Text semantic vector: including semantic vectors, keywords, and semantic tags. The preprocessed text semantic data is input into the BERT model to generate a 768-dimensional vector, and keywords (such as "currency fund growth") are extracted. The cosine similarity between different text vectors is then calculated.

[0089] The BERT model, based on the Transformer architecture, transforms input text into a 768-dimensional semantic vector through bidirectional contextual learning. This vector incorporates the text's semantic features, contextual information, and relationships between vocabulary, providing a deep numerical representation of the text's semantics. For example, for the text "balance sheet," the value of each dimension in the 768-dimensional vector reflects its degree of association with semantics such as "financial statements" and "capital flow," as well as its specific meaning within the document context.

[0090] In subsequent steps, by calculating the cosine similarity between 768-dimensional vectors, the semantic relevance between texts can be determined. For example, in multimodal feature fusion and conflict resolution, by comparing the vector similarity between "balance sheet" and the text in the table area, the logical association between them can be determined, assisting in determining whether the table content falls under that heading. Combined with tools such as spaCy, the 768-dimensional vector can enhance the recognition accuracy of entities in the text (such as "monetary funds" marked as FINANCIAL_ENTITY), providing support for building the semantic structure of the document. When processing scanned text, if there are errors in OCR recognition, the semantic vector generated by BERT can be used as one of the judgment bases, combined with visual modal information to perform cross-modal conflict resolution and correct erroneous text recognition results.

[0091] Based on BERT semantic vectors and combined with TF-IDF or TextRank algorithms, high-frequency or high-weight words (such as "balance sheet" and "opening balance") in the text are extracted as keywords.

[0092] Named entity recognition is performed using spaCy to mark entity types in the text (such as financial entities, dates, and locations), and the semantic hierarchy is constructed by combining the font size extracted by PDFMiner (such as ≥18pt is marked as Title).

[0093] (2) Visual feature vector: Call a lightweight YOLOX model (such as yolox-nano.onnx) to infer each pixel sub-block in the preprocessed image data, and output the bounding box coordinates (x1, y1, x2, y2), category label (such as Table, Title, Footnote, Logo) and confidence score (such as 0.92) of each detected target as the visual feature vector of the pixel sub-block. The visual feature vectors of each pixel sub-block are spliced ​​into a matrix to obtain the final visual feature vector.

[0094] The output of the YOLOX model is not only the key visual information for multimodal fusion, but also directly affects strategy selection and structured analysis results, forming a technical chain of "image preprocessing → visual detection → feature fusion → structured output", ensuring the accurate recognition and positioning of complex elements such as tables and images in documents.

[0095] Furthermore, when the YOLOX model detects a specific visual element (such as a table or title area), it performs directional OCR recognition on that area. The bounding box coordinates output by the YOLOX model are used to locate the pixel sub-block that needs to be recognized. The Tesseract OCR engine is called to perform high-precision recognition on the area to obtain the text content and confidence score. The text content recognized by OCR (such as "Currency") is combined with the regional category of the visual modality ("TableTitle"), and cross-modal error correction is performed through semantic constraints (financial dictionary) and coordinate space relationships (located in the key area of ​​the financial statement). The text coordinates recognized by OCR must be consistent with the coordinates of the visual element. For example, the coordinates of "Currency") must be located within the table title area detected by YOLOX, otherwise it may be judged as a misrecognition.

[0096] (3) Coordinate feature vector: This includes the normalized coordinates of text blocks and tables to quantify spatial positional relationships. The coordinates output by YOLOX are based on pixel space. After conversion by the rect_to_bbox function, they are mapped to the PDF point space coordinates and normalized to the [0, 1] interval to facilitate subsequent multimodal feature fusion (such as combining with text coordinates and semantic features).

[0097] Specifically, the pixel coordinates are scaled to the normalized range required by the model input (for example, YOLOX requires the coordinate range to be [0,1]). The formula is: , ,in, is the normalized x-axis coordinate, is the normalized y-axis coordinate. The normalized coordinate of the upper left corner of the sub-block is: ;The normalized coordinates of the lower right corner of the sub-block are: Example result: The normalized coordinates of the pixel coordinate (10,10,20,20) are (0.0125,0.0167,0.025,0.0333).

[0098] Quantifying spatial relationships involves converting the relative positions of elements within a PDF document (such as text blocks, tables, and images) into calculable numerical information. This allows computers to accurately understand the physical layout of elements, such as "who is above whom," "who is close to whom," and "who contains whom." This provides the data foundation for logical hierarchical analysis, multimodal information fusion, and structured output.

[0099] The specific process of quantifying spatial position relationships is as follows.

[0100] 1) Unify and normalize the coordinate system: First, the coordinate system in the PDF document is unified to the image's pixel coordinates (with the upper-left corner as the origin). For example, point coordinates in the PDF are converted to pixel coordinates, ensuring that all element positions are based on the same reference system. Pixel coordinates are then scaled to the interval [0, 1] to ensure consistent comparison of element positions across pages of different sizes. For example, if an element's horizontal position on a page is 100 pixels, and the total width of the page is 800 pixels, its normalized horizontal coordinate is 100 / 800 = 0.125.

[0101] 2) Calculate the distance between elements: Using normalized coordinates, calculate the vertical (Y-axis) and horizontal (X-axis) distances between two elements. For example, if the vertical coordinate range of element A is 0.2-0.3, and the vertical coordinate range of element B is 0.35-0.4, then the vertical distance between elements A and B is 0.35-0.3 = 0.05. Set a distance threshold (e.g., a vertical distance less than 0.08 corresponds to 50 pixels). When the distance between two elements is less than the threshold, they may be considered to have a logical connection (e.g., a title and a table below).

[0102] 3) Determine the orientation of elements: If the right normalized coordinate of element A is smaller than the left normalized coordinate of element B, A is to the left of B. For example, if the horizontal coordinate of element A is 0.1-0.3 and that of element B is 0.4-0.6, then element A is to the left of element B. If the bottom normalized coordinate of element A is smaller than the top normalized coordinate of element B, then element A is above element B. For example, if the vertical coordinate of element A is 0.2-0.3 and that of element B is 0.4-0.5, then element A is above element B.

[0103] 4) Determine the inclusion relationship of elements: If the coordinate range of element A completely contains the coordinate range of element B (that is, A's left boundary ≤ B's left boundary, A's right boundary ≥ B's right boundary, A's upper boundary ≤ B's upper boundary, and A's lower boundary ≥ B's lower boundary), then element A is considered to contain element B. For example, if the coordinate range of a large table contains the coordinates of a cell, then the cell is part of the table.

[0104] This application can match semantic information with visual layout by quantifying the positional relationship of elements such as text, tables, and images. For example, the text "Balance Sheet" is spatially adjacent to the table on the right, and it can be inferred that they are related in content, thereby improving the accuracy of multimodal fusion. Further based on the distance and orientation between elements, their logical relationship is inferred. For example, the title and the table 28 pixels below are judged to be a parent-child relationship of "title-table" because the distance is less than 50 pixels; multiple columns of text are arranged in order from left to right, which can restore the reading order. The quantified coordinate information can accurately record the position of elements in the document (with an error of less than 5 pixels). This data is attached to the structured output (such as JSON) as metadata to support subsequent visual verification and document restoration. For example, the parsed table coordinates can be used directly for page rendering to ensure consistency with the original PDF layout.

[0105] Step 63: adopt a trimodal cross-attention mechanism to perform cross-modal attention fusion on the text semantic vector, the visual feature vector and the coordinate feature vector to quantify the semantic-spatial correlation between the text and the visual elements.

[0106] Among them, the formula of the trimodal cross attention mechanism is: The meaning and examples of each parameter are shown in Table 3.

[0107] Table 3 Parameter meaning and examples

[0108] In step 63, the visual feature vector and the coordinate feature vector are first concatenated by dimension to form a joint feature matrix The dot product of the text semantic vector and the transposed joint feature matrix is ​​then performed to calculate the initial correlation (for example, the dot product of the text "balance sheet" and the table feature is 0.72). This indicates a 72% semantic-spatial match between the text and the table area. The correlation is then normalized into an attention score using the Softmax function. The attention score for the table area increases to 0.91, guiding the OCR engine to prioritize this highly correlated area and avoid redundant computation during a full image scan.

[0109] By calculating the attention score between the text semantic vector and the visual feature vector, the correlation between the two is quantified (value ∈ [0, 1]), prioritizing highly correlated areas. For example, if the attention score between the text "balance sheet" and the table area detected by YOLOX is 0.91, indicating a very high semantic-spatial match between the two, OCR analysis of this table area is prioritized, avoiding redundant computations required for full-image scanning and improving efficiency by 30%.

[0110] The attention score serves as a weighting factor for multimodal fusion, adjusting the contributions of text, visual, and coordinate modalities. For example, when OCR recognition results conflict with visual localization (e.g., misidentification of "currency"), if the visual region has a higher attention score (e.g., 0.92), the visual modality weight (0.6) will dominate the error correction. Combined with the financial dictionary, the result will be corrected to "currency," improving text accuracy by 12.5%.

[0111] The attention score guides the OCR engine to prioritize high-correlation areas through cross-modal correlation quantification and dynamic region prioritization. The specific mechanism and processing logic are as follows.

[0112] (1) The correlation between the text and each visual element is calculated by performing a dot product operation on the text semantic vector, the visual feature vector, and the coordinate feature vector. The attention score is then obtained after Softmax normalization. For example, the dot product of the BERT semantic vector of the text "balance sheet" and the table area features detected by YOLOX shows an initial correlation of 0.72. After Softmax, the attention score increases to 0.91, indicating that the semantic-spatial match between the text and the table area is extremely high.

[0113] (2) The OCR engine (such as Tesseract) sorts the visual element areas from high to low according to the attention score, and gives priority to parsing high-score areas (such as the table title area with an attention score > 0.8) to avoid redundant calculations of the entire image scan.

[0114] (3) Refined OCR (such as 300DPI image segmentation and perspective correction) is used for high-score areas, and fast extraction or skipping strategies are used for low-score areas (such as background images with attention scores < 0.5).

[0115] Specifically, the processing flow for highly correlated regions is as follows: A trimodal cross-attention mechanism is used to generate an attention score for each visual element. Elements are sorted in descending order by score, forming a parsing queue (e.g., table → title → footnote). Tesseract is used for targeted recognition of high-scoring regions, and semantic constraints (such as financial dictionaries) are combined to improve accuracy. For example, in the parsing of a financial report table, the attention score between the table title "Balance Sheet" and the table area is 0.91. OCR prioritizes parsing this area, extracting the table structure and data using the layout_resolve strategy, improving accuracy to 95.1%. An HTML table with semantic tags is generated, and coordinate metadata is recorded.

[0116] The processing flow for low-correlation regions is as follows: If the region's attention score is <0.5 and text extractability is True (pdf_text_extractable=True), PDFMiner is used to extract the text quickly, skipping OCR. This reduces the time required to extract the text from 12.7s / page to 8.3s / page, improving efficiency by 34.6%. If the region is a scanned document and the attention score is low, pure OCR is used to extract the text without performing detailed layout analysis. This is a fallback solution that covers over 95% of document types. For example, a background logo region with an attention score of 0.23 is considered non-critical content. Only image coordinates and categories are extracted, without performing OCR text recognition, reducing inefficient computation and focusing on core data analysis.

[0117] By dynamically allocating computing resources through attention scores, high-correlation areas are refined (with 95.1% accuracy) and low-correlation areas are rapidly processed (reducing time by 34.6%). Strategies are automatically switched based on the attention score, eliminating the need for manual intervention and adapting to over 95% of document types, including text, scanned documents, and documents with complex layouts. Attention scores are recorded as metadata in JSON output, supporting parsing result verification and strategy optimization. Attention scores, quantified through a dual "semantic-spatial" correlation, enable intelligent scheduling of the OCR engine, addressing the efficiency bottlenecks and complex layout parsing errors of traditional full-image parsing. This approach is particularly suitable for scenarios requiring high document parsing accuracy, such as finance and law.

[0118] Step 64, based on the semantic-spatial correlation, the text semantic vector, the visual feature vector and the coordinate feature vector, a pre-trained neural network model is used to dynamically adjust the weights of text, vision and coordinates to obtain text weight, visual weight and coordinate weight.

[0119] The neural network model is an error correction network, a lightweight neural network used to calculate modal weights when fusing multimodal features, resolving conflicts between text, visual, and coordinate features (such as semantic inconsistencies between OCR errors and visual localization). The error correction network is a fully connected neural network with a simple and efficient structure. It eliminates the need for complex convolutional or recurrent structures, directly processes vector inputs and weighted outputs, and is capable of adaptive weight learning.

[0120] The principle of the error correction network to achieve the error correction function is as follows.

[0121] (1) Multimodal evidence fusion: When there is a conflict between the text modality and the visual / coordinate modality (e.g., the OCR result is semantically unreasonable but the visual positioning is clear), the error correction network resolves the conflict by adjusting the weights. The modality with higher confidence is given a higher weight. For example, if the visual modality confidence is 0.92 > the text modality confidence is 0.65, then a higher weight is assigned. The coordinate modality verifies the spatial proximity of the text and the table, then a weight is assigned to strengthen the visual decision.

[0122] (2) Semantic Constraints and Dynamic Adaptation: Domain dictionaries (e.g., financial terms) are incorporated into network training. When the edit distance between the OCR result and the dictionary item is ≤1 and the visual weight is >0.5, the error correction logic is triggered. The weight is dynamically adjusted based on the document area type. For example, in the table title area, the visual weight is automatically increased (≥0.5) and the text weight is decreased (≤0.4) because the visual positioning (e.g., border, position) of the table title is more reliable than the text content.

[0123] Among them, the area types include: table title area, text area, footer area, header / footer area, image area and list area. The visual modality weight of the table title area is higher, because the position of the table title (such as fixed above the table) and visual features (such as larger font and border) are easier to accurately detect through the YOLOX model than text content, and are more reliable. In the text area, the weights of text and visual modalities are relatively balanced (such as 0.5:0.5), because the content of the text relies on the semantic understanding of the text and the visual layout (such as paragraph spacing and font style). The weight of coordinate modality in the footnote area is increased, because the spatial position of the footnote (bottom of the page) is the main basis for identification, and the importance of visual and text modalities is relatively reduced. In the image area, the visual modality weight is dominant, and the text modality weight is relatively low, because the type of image elements (such as LOGO, chart) is mainly determined by visual detection.

[0124] (3) End-to-end training optimization: The cross-entropy loss function is used, with the difference between the parsed results and the manual annotations as the optimization target. The weight matrix is ​​optimized through the back-propagation algorithm, so that the model prioritizes high-confidence modalities in conflict scenarios, thereby improving the error correction accuracy. For example, after training, the text accuracy of the OCR error correction scenario increased from 87.5% to 100%, and the confidence level increased from 0.65 to 0.812.

[0125] Among them, the parsing result is the text content, confidence score, element category, and relevant metadata output by the error correction network through multi-modal fusion and error correction. Among them, the text content is the text recognized by OCR and corrected through cross-modal error correction (such as correcting "货币姿金" to "货币资金"). The confidence score is the reliability score of the result after fusing multi-modal information. The relevant metadata includes metadata such as the spatial coordinates and hierarchical relationships of elements (such as the parent-child element IDs). Annotation refers to the correct text, element type, and hierarchical relationship generated manually or through a standard process, which serves as the supervision signal for model training, including: correct text label (accurate text content in the original document), true element category, and true hierarchical relationship (logical association between elements).

[0126] Traditional OCR only relies on image features and is vulnerable to font and noise effects (such as misrecognizing "资" as "姿"). The error correction network realizes cross-modal collaborative error correction by fusing visual positioning (table title area) and coordinate proximity (spatial association with the table), combined with semantic constraints (financial dictionary). In the OCR recognition and error correction case, when Tesseract misrecognizes "货币资金" as "货币姿金", the error correction network outputs weights, and the visual modality weight is significantly higher than the text modality. Finally, the fused feature is dominated by 60% of the visual features, and combined with the semantic constraints of the table title, it is automatically corrected to "货币资金". This fused feature not only corrects the error but also improves the confidence score, from 0.65 to 0.812, enhancing the consistency of text with visual and coordinate information.

[0127] In documents with complex layouts such as financial reports and legal contracts, the error correction network can ensure that the extraction accuracy of table data is increased from 65.2% to 95.1%, and the parsing time of complex documents is reduced by 34.6% (from 12.7 s / page to 8.3 s / page). For example, under the action of the error correction network, "货币姿金" is corrected to "货币资金" through the dominance of visual weight and semantic dictionary matching, and the confidence after fusion is increased by 24.9% (from 0.65 to 0.812).

[0128] In summary, the error correction network maps multi-modal features to dynamic weights through a fully connected neural network structure, realizing an error correction mechanism of "dominance of high-confidence modalities and collaboration of multi-modal evidence", effectively solving the error problem of traditional single-modal parsing, which is one of the core innovation points for improving the parsing accuracy of unstructured PDFs.

[0129] Step 65, according to the text weight, the visual weight, and the coordinate weight, perform weighted summation on the text semantic vector, the visual feature vector, and the coordinate feature vector to obtain a multi-modal fusion feature.

[0130] Specifically, use the formula and Get multimodal fusion features. Among them, It is a trimodal concatenation vector, which concatenates the text semantic vector, visual feature vector and coordinate feature vector in sequence (total dimension 1796) as the input of the error correction network. It is a learnable weight matrix that optimizes the modal weight mapping relationship through training. The three output nodes correspond to the weight parameters of text, vision, and coordinates respectively. is the bias vector, which enhances the model's ability to fit nonlinear weight relationships. It is a Sigmoid activation function that compresses the output to the interval [0,1] and generates a modal weight vector. , respectively represent the contribution of text, vision, and coordinate modalities in the fusion, namely text weight, vision weight, and coordinate weight. It is an element-by-element weighted operation, and each modal feature is dynamically weighted and summed through the modal weight to generate the multimodal fusion feature F.

[0131] In an exemplary embodiment, a typical conflict resolution scenario is provided: Tesseract misidentifies "currency funds" as "currency posture gold" (text confidence is 0.65), but YOLOX detects that the text is located in the table title area (visual confidence is 0.95, coordinate confidence is 0.98). The error correction mechanism outputs weights G = [0.2, 0.6, 0.2], and the visual modality weight is significantly higher than the text modality. The multimodal fusion feature F is dominated by 60% of visual features and combined with the semantic constraints of the table title, automatically correcting it to "currency funds", achieving cross-modal collaborative error correction.

[0132] In a specific application example, generating the third structured data according to the multimodal fusion features in step 6 includes the following steps 66 to 67.

[0133] Step 66: perform element classification and coordinate metadata addition on the multimodal fusion features, and convert the multimodal fusion features into structured elements with semantic labels.

[0134] The semantic consistency of elements is verified through attention scores. For example, if the attention score between a text block and a table area is greater than 0.8, the text is classified as a table title (marked as "TableTitle") to avoid classification errors caused by OCR misrecognition. Furthermore, the attention score is combined with spatial proximity to construct parent-child relationships between elements. For example, if the title "Balance Sheet" has an attention score of 0.91 with the table, and the vertical distance is 28 pixels < 50 pixels, a hierarchical association of "Title→Table" is established and recorded in the JSON output through the parent field (such as "parent":"Title_001"). The physical order of elements is restored based on spatial coordinates, and a logical hierarchy is established through semantic association to ensure that the structured results conform to human reading habits.

[0135] The process of element classification is as follows.

[0136] (1) Text element classification: Use spaCy to perform semantic boundary segmentation on the searchable text stream, combined with the font size (e.g. ≥18pt is marked as Title), spatial position (Y < 20% of the page height is the top title) and regular expression (e.g. Identify footnotes), generate Text elements and mark semantic tags (Title / Body / Footnote). If the text has been corrected (such as "currency" → "currency"), the corrected text content will prevail and the confidence score will be updated. Example: The cover title "2024 Q3 Financial Report" is marked as Title type because the font size is ≥18pt and it is located at the top of the page; the footnote "Data as of September 30, 2024" is marked as " ” symbol and is located at the bottom of the page, labeled Footnote.

[0137] The final output is a structured Text element with attached semantic tags (Title / Body / Footnote), coordinate metadata (PDF point and pixel coordinates), and font style information. For example, the cover title "2024 Q3 Financial Report" is labeled as Title, with coordinates recorded as (X1=100, Y1=272, X2=200, Y2=292) and the spatial attribute "centered at the top of the page" noted.

[0138] (2) Classification of table elements: Based on the table bounding box detected by the YOLOX model (e.g., x1=150, y1=300), TesseractOCR is called to perform directional recognition of the cell area. Rows are segmented by vertical coordinate spacing (e.g., row spacing > 20 pixels), and columns are segmented by horizontal coordinate spacing combined with semantic labels (e.g., "subject," "amount") to infer the relationship between the table header and the data row. An HTML table with semantic labels is generated. Example: After the detected table area is recognized by OCR, the balance sheet structure is generated based on the row and column segmentation results and marked as Table type.

[0139] The final output is a structured Table element: containing HTML-formatted table data, row and column semantic inference results (such as table headers and data rows), and coordinate metadata. For example, after the table area detected by the YOLOX model (X1=150, Y1=300, X2=550, Y2=500) is targeted by Tesseract, an HTML table with "Subject" and "Amount" columns is generated, and the row structure is segmented by vertical spacing (30 pixels > 20 pixels threshold).

[0140] (3) Image element classification: Generate an Image element based on the classification results of the YOLOX model, and record the pixel space coordinates (after rect_to_bbox calibration). Record the image format (PNG / JPEG), coordinates, and category label, such as {"type":"Image","category":"Logo"}. Example: An embedded logo is detected by YOLOX as belonging to the "Logo" category, and an Image element is generated and the coordinates are recorded.

[0141] In a specific application example, coordinate standardization is first performed, using pixel-space coordinates (with the origin in the upper left corner and increasing Y-axis downwards) to unify the coordinate base of all elements. Two-dimensional sorting rules prioritize the X-axis, sorting elements in ascending order by their left boundary coordinate (X1), ensuring left-to-right order (e.g., multi-column text sorts in the left column first). The Y-axis takes precedence, sorting elements in the same column in ascending order by their top boundary coordinate (Y1), ensuring top-to-bottom order (e.g., title → table → footnote). When the X1 difference between elements exceeds the column width threshold (default 50 pixels), they are identified as separate columns, sorted separately, and then merged (e.g., left-column text and right-column footnotes sort independently). If the vertical distance between a child element and its parent element is less than 50 pixels and they belong to the same page, a parent-child relationship is established. For example, if the vertical distance between a table title (Y=272) and a table area (Y=300) is 28 pixels and less than 50 pixels, a "title-table" parent-child relationship is established. Semantic validation, combined with element type tags (e.g., the logical association between Title and Table), improves the accuracy of hierarchical determination.

[0142] Metadata appending refers to the addition of additional data describing the properties, relationships, and characteristics of each element (such as text, table, and image) when a PDF document is parsed into structured elements. This data does not directly relate to the element's content (such as the specific text in a text file or the numerical values ​​in a table). Instead, it describes the element's role, location, and relationships within the document. This data is stored in a metadata field in JSON format.

[0143] The process of attaching coordinate metadata is as follows.

[0144] (1) Coordinate system conversion and normalization: Use the rect_to_bbox function to convert the PDF point to the original point in the lower left corner and the image point in the upper left corner. Divide the pixel coordinates by the image width / height and scale them to the range [0,1]. For example: PDF point coordinates (100pt,520pt) are converted to pixel coordinates (416.67px,216.67px). When the image width is 800px and the height is 600px, the normalized coordinates are (0.5208,0.3611).

[0145] (2) Coordinate metadata attachment rules: Text elements are attached with font size, level labels, and pixel / point space coordinates; table / image elements are attached with bounding box coordinates and normalized coordinates detected by YOLOX.

[0146] The following effects can be achieved by appending metadata.

[0147] 1) Build a logical document hierarchy, using the parent and children fields to clearly define the logical relationships between elements. For example, a table can be associated with its title as a parent element and footnotes as children, forming a "Title → Table → Footnote" hierarchy. This facilitates logical navigation and understanding of the document. For example, metadata:{"parent":"Title_001","children":["Footnote_003"]} indicates that the table is associated with the title "Title_001" and has the footnote "Footnote_003" as a child element.

[0148] 2) Ensure that elements are ordered. The sort_order field records the order in which elements appear in the document, ensuring that the structured output conforms to human reading habits (such as left to right and top to bottom). For example, sort_order:2 indicates that the element is the second main content block in the document.

[0149] 3) Support for result verification and traceability: Storing metadata such as element parsing confidence and feature weights facilitates subsequent verification of the reliability of parsing results. For example, the metadata for a table element can record its visual inspection confidence (0.95) and its confidence after text correction (0.812).

[0150] 4) Enhance the semantics of structured data: Add semantic categories of additional elements (such as "BalanceSheet"), coordinate spaces (such as pixel_space), format information (such as font size), etc., so that structured data has richer semantic information and is suitable for AI downstream applications.

[0151] This application uses multimodal feature fusion to convert unstructured elements such as tables and images in PDFs into structured elements with semantic labels, addressing the structural confusion inherent in traditional single-modal parsing. For example, a table in a scanned PDF document was generated into a computable Table element using YOLOX positioning, OCR recognition, and semantic constraints, improving accuracy from 65% to 95.1%. This structured representation of unstructured information transforms previously unstructured text and images in PDFs into standardized, computer-understandable elements, breaking the limitations of traditional parsing, which can only extract text streams or image pixels. Furthermore, by combining BERT semantic vectors with coordinate spatial relationships, it can understand the document's logical hierarchy (e.g., title → body → footnotes), rather than simply parsing at the pixel level. For example, the "cash and cash equivalents" text in a financial report and the amount area in the table are correctly classified as Table-related elements through a dual semantic-spatial association. The generated structured Table element (e.g., a balance sheet) can directly extract data such as "cash and cash equivalents: opening balance of RMB 50 million," reducing manual entry errors and adapting to the needs of high-frequency financial report parsing. Through hierarchical construction (e.g., clause paragraph → appendix chart), key clauses in contracts can be quickly located, improving document review efficiency by over 30%. After processing the scanned document in step 66, a Text / Table element with coordinate metadata is generated, supporting subsequent automated retrieval and structured management. Image and Text elements are spatially aligned using coordinate metadata. For example, the text "LOGO" is associated with the corresponding Image element in terms of coordinates, forming a "text-image" semantic pair.

[0152] Step 67: Fill the JSON template with the structured elements to obtain third structured data. For example, the HTML structure of the Table element is converted to the "html" field. The element's "metadata" field (such as coordinates and weight) is mapped to the "metadata" block.

[0153] In a specific application example, the decision-making process is first visualized: attention scores are stored as metadata in the metadata.attention_scores field of the JSON file, recording the intermediate results of multimodal fusion and facilitating subsequent verification and error correction. The output is then optimized: attention scores guide element sorting and layout restoration. For example, elements are sorted in descending order by attention score, ensuring that key content (such as highly relevant title-table pairs) is prioritized for output, consistent with human reading habits. Table elements with HTML structure are output when tables are included. Text elements and semantic tags are output when plain text is included.

[0154] The predefined structure of the JSON template is shown in Table 4.

[0155] Table 4 Core field template description

[0156] The data mapping rules for splicing multimodal data into JSON are shown in Table 5.

[0157] Table 5 Data mapping rules

[0158] Compared with the prior art, this application has the following beneficial effects.

[0159] (1) Improved table recognition accuracy: OpenCV edge detection + YOLOX object detection locates the table area, ensuring accurate recognition of high-confidence tables. YOLOX outputs the bounding box coordinates and category labels of the table, providing accurate visual positioning. The trimodal cross-attention mechanism fuses text semantics (BERT vectors), visual features (YOLOX detection results), and coordinate features to strengthen the semantic-spatial association of the table area. The dynamic weighted error correction mechanism prioritizes high-confidence visual modalities (e.g., table detection confidence of 0.95) to correct OCR recognition errors and improve the accuracy of table content extraction.

[0160] (2) Reduced parsing efficiency for complex documents: This strategy can quickly determine whether a document is searchable text. If pdf_text_extractable=True, it skips OCR and directly enters the fast_text_extract strategy. It automatically switches strategies based on layout complexity (such as whether it contains tables), avoiding redundant layout analysis for simple documents. By focusing on highly relevant areas through attention scores, it only performs refined parsing on key content (such as table titles), reducing the computational effort of OCR.

[0161] (3) Improved coverage of document types: Through the two-tier decision-making of the auto strategy (text extractability → layout complexity), documents are divided into four scenarios (text type without tables, text type with tables, scanned documents without tables, and scanned documents with tables). Each scenario is matched with the optimal strategy, avoiding the limitations of a single strategy. Regardless of whether the document can extract text, it can be parsed based on text semantics (if available) or visual features (scanned documents). For example, scanned documents are processed using YOLOX + OCR, and text-type documents are processed using PDFMiner + semantic analysis, ensuring the feasibility of parsing all types of documents.

[0162] (4) Enhanced traceability: By appending decision-making process data to the structured output, each step of the parsing result can be traced. For example, the confidence level of 0.812 for the OCR error correction result "monetary funds" is calculated by visual weight 0.6×0.92 + text weight 0.4×0.65, and users can use this to verify the rationality of the result. The explicit recording of the relationship between text semantics, visual positioning, and coordinate space facilitates verification of the consistency of multimodal information during manual review (such as whether the spatial proximity of text content and table area matches).

[0163] Based on the same inventive concept, embodiments of the present application also provide a PDF document intelligent parsing device for implementing the aforementioned method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more embodiments of the PDF document intelligent parsing device provided below can be found in the above-described limitations on the method and will not be further elaborated here.

[0164] In an exemplary embodiment, Figure 3 As shown, a PDF document intelligent parsing device is provided, including: a preprocessing module 301, a text judgment module 302, a first parsing module 303, a table judgment module 304, a second parsing module 305 and a third parsing module 306.

[0165] The pre-processing module 301 is used to obtain a PDF document to be parsed, parse the page elements in the PDF document, and generate a document metadata dictionary, which includes the number of pages and element distribution.

[0166] The text determination module 302 is configured to determine whether the PDF document contains extractable text based on the document metadata dictionary.

[0167] The first parsing module 303 is configured to convert the PDF document into an image and perform optical character recognition to generate first structured data when the PDF document does not contain extractable text.

[0168] The table determination module 304 is configured to determine whether the PDF document contains a table by using traditional visual methods and deep learning algorithms when the PDF document contains extractable text.

[0169] The second parsing module 305 is configured to extract text using PDFMiner to generate second structured data when the PDF document does not contain a table.

[0170] The third parsing module 306 is configured to, when the PDF document contains a table, perform multimodal feature extraction and feature fusion on the PDF document based on the document metadata dictionary to obtain multimodal fusion features, and generate third structured data based on the multimodal fusion features. The first structured data, the second structured data, or the third structured data is the result of parsing the PDF document.

[0171] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0172] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0173] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0174] In this application, all actions to obtain signals, information or data are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0175] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for intelligent parsing of unstructured PDF documents, characterized in that: The method comprises: Obtain a PDF document to be parsed, parse the page elements in the PDF document, and generate a document metadata dictionary; the document metadata dictionary includes the number of pages and element distribution; Determining whether the PDF document contains extractable text based on the document metadata dictionary; If the PDF document does not contain extractable text, converting the PDF document into an image and performing optical character recognition to generate first structured data; If the PDF document contains extractable text, a traditional visual method and a deep learning algorithm are used to determine whether the PDF document contains a table. Specifically, the method includes: using an OpenCV edge detection algorithm and a Hough transform to screen horizontal lines in the PDF document and calculate the horizontal line density; using a deep learning algorithm to perform object detection on the unstructured PDF document and determine a table detection result; the table detection result includes a table bounding box, a confidence score, and a category label; and determining whether the PDF document contains a table based on the horizontal line density and the table detection result. If the PDF document does not contain a table, extracting text using PDFMiner to generate second structured data; If the PDF document contains a table, multimodal feature extraction and feature fusion are performed on the PDF document according to the document metadata dictionary to obtain multimodal fusion features, and third structured data is generated based on the multimodal fusion features; wherein the first structured data, the second structured data, or the third structured data is the parsing result of the PDF document.

2. The unstructured PDF document intelligent parsing method according to claim 1, characterized in that: Parse the page elements in the PDF document and generate a document metadata dictionary, specifically including: The pdfminer.high_level.extract_pages() function in Python is called to parse the PDF document page by page to determine the number of pages and element distribution of the PDF document to generate a document metadata dictionary.

3. The unstructured PDF document intelligent parsing method according to claim 1, characterized in that: Determining whether the PDF document contains extractable text based on the document metadata dictionary specifically includes: Based on the document metadata dictionary, the pdfminer.high_level.extract_text() function in Python is called to extract the text stream. If the text stream extraction is successful, it is determined that the PDF document contains extractable text; otherwise, it is determined that the PDF document does not contain extractable text.

4. The unstructured PDF document intelligent parsing method according to claim 1, characterized in that: According to the document metadata dictionary, multimodal feature extraction and feature fusion are performed on the PDF document to obtain multimodal fusion features, specifically including: According to the document metadata dictionary, the PDF document is subjected to text semantic tagging, image block calibration and coordinate calibration processing to obtain preprocessed text semantic data, preprocessed image data and preprocessed coordinate data; Performing feature extraction on the preprocessed text semantic data, the preprocessed image data, and the preprocessed coordinate data to obtain a text semantic vector, a visual feature vector, and a coordinate feature vector; A trimodal cross-attention mechanism is used to perform cross-modal attention fusion on the text semantic vector, the visual feature vector, and the coordinate feature vector to quantify the semantic-spatial correlation between the text and the visual elements. According to the semantic-spatial correlation, the text semantic vector, the visual feature vector and the coordinate feature vector, a pre-trained neural network model is used to dynamically adjust the weights of text, vision and coordinates to obtain text weight, visual weight and coordinate weight; According to the text weight, the visual weight and the coordinate weight, a weighted sum is performed on the text semantic vector, the visual feature vector and the coordinate feature vector to obtain a multimodal fusion feature.

5. The unstructured PDF document intelligent parsing method according to claim 4, characterized in that: Feature extraction is performed on the preprocessed text semantic data, the preprocessed image data, and the preprocessed coordinate data to obtain a text semantic vector, a visual feature vector, and a coordinate feature vector, specifically including: Using the BERT model to extract text semantic vectors of the preprocessed text semantic data; Using the YOLOX model to extract visual features from the preprocessed image data to obtain a visual feature vector; The pre-processed coordinate data is normalized to obtain a coordinate feature vector.

6. The unstructured PDF document intelligent parsing method according to claim 1, characterized in that: Generating third structured data according to the multimodal fusion features specifically includes: Performing element classification and coordinate metadata addition on the multimodal fusion features, and converting the multimodal fusion features into structured elements with semantic labels; The structured elements are used to fill in the JSON template to obtain third structured data.

7. An intelligent parsing device for unstructured PDF documents, characterized in that: The device is applied to the unstructured PDF document intelligent parsing method according to any one of claims 1 to 6, and the device includes: A preprocessing module is used to obtain a PDF document to be parsed, parse the page elements in the PDF document, and generate a document metadata dictionary; the document metadata dictionary includes the number of pages and element distribution; A text determination module, configured to determine whether the PDF document contains extractable text based on the document metadata dictionary; A first parsing module is configured to convert the PDF document into an image and perform optical character recognition to generate first structured data when the PDF document does not contain extractable text; A table determination module is used to determine whether the PDF document contains a table by using traditional visual methods and deep learning algorithms when the PDF document contains extractable text; A second parsing module is configured to extract text using PDFMiner to generate second structured data when the PDF document does not contain a table; a third parsing module configured to, when the PDF document contains a table, perform multimodal feature extraction and feature fusion on the PDF document according to the document metadata dictionary to obtain multimodal fusion features, and generate third structured data based on the multimodal fusion features; wherein the first structured data, the second structured data, or the third structured data is a parsing result of the PDF document.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the unstructured PDF document intelligent parsing method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for intelligent parsing of unstructured PDF documents according to any one of claims 1 to 6 is implemented.

Images